Release Notes - Kafka - Version 2.7.0

Below is a summary of the JIRA issues addressed in the 2.7.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-9893] - Configurable TCP connection timeout and improve the initial metadata fetch
[KAFKA-9915] - Throttle Create Topic, Create Partition and Delete Topic Operations
[KAFKA-10054] - Add TRACE-level end-to-end latency metrics to Streams
[KAFKA-10259] - KIP-554: Add Broker-side SCRAM Config API
[KAFKA-10338] - Support PEM format for SSL certificates and private key

Improvement

[KAFKA-5636] - Add Sliding-Window support for Aggregations
[KAFKA-6585] - Consolidate duplicated logic on reset tools
[KAFKA-6733] - Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
[KAFKA-7334] - Suggest changing config for state.dir in case of FileNotFoundException
[KAFKA-8360] - Docs do not mention RequestQueueSize JMX metric
[KAFKA-8836] - Add inter-broker protocol to alter ISR
[KAFKA-9018] - Kafka Connect - throw clearer exceptions on serialisation errors
[KAFKA-9076] - MirrorMaker 2.0 automated consumer offset sync
[KAFKA-9194] - Missing documentation for replicaMaxWaitTimeMs config value
[KAFKA-9439] - Add more public API tests for KafkaProducer
[KAFKA-9450] - Decouple inner state flushing from committing
[KAFKA-9587] - Producer configs are omitted in the documentation
[KAFKA-9924] - Add RocksDB Memory Consumption to RocksDB Metrics
[KAFKA-9929] - Support reverse iterator on WindowStore
[KAFKA-10002] - Improve performances of StopReplicaRequest with large number of partitions to be deleted
[KAFKA-10020] - KIP-616: Rename implicit Serdes instances in kafka-streams-scala
[KAFKA-10023] - Enforce broker-wide and per-listener connection creation rate (KIP-612, part 1)
[KAFKA-10033] - AdminClient should throw UnknownTopicOrPartitionException instead of UnknownServerException if altering configs of non-existing topic
[KAFKA-10035] - Improve the AbstractResetIntegrationTest
[KAFKA-10040] - Make computing the PreferredReplicaImbalanceCount metric more efficient
[KAFKA-10077] - Filter downstream of state-store results in spurious tombstones
[KAFKA-10098] - Remove unnecessary escaping in regular expression in SaslAuthenticatorTest.java
[KAFKA-10106] - log time taken to handle LeaderAndIsr request
[KAFKA-10122] - Consumer should allow heartbeat during rebalance as well
[KAFKA-10126] - Deprecate unused options in ConsumerPerformance
[KAFKA-10131] - Minimize use of --zookeeper flag in ducktape tests
[KAFKA-10135] - Extract Task#executeAndMaybeSwallow to be a general utility function into TaskManager
[KAFKA-10141] - Add more detail to log segment deletion message
[KAFKA-10143] - Add integration testing ensuring that the reassignment tool can change replication quota with rebalance in progress
[KAFKA-10168] - Rename public StreamsConfig variable
[KAFKA-10186] - Aborting transaction with pending data should throw non-fatal exception
[KAFKA-10205] - NullPointerException in StreamTask
[KAFKA-10211] - Add DirectoryConfigProvider
[KAFKA-10225] - Increase default zk session timeout for system tests
[KAFKA-10232] - MirrorMaker2 internal topics Formatters
[KAFKA-10277] - Relax non-null key requirement for KStream-GlobalKTable joins
[KAFKA-10279] - Allow dynamic update of certificates with additional SubjectAltNames
[KAFKA-10305] - Print usage when parsing fails for ConsumerPerformance
[KAFKA-10316] - Consider renaming getter method for Interactive Queries
[KAFKA-10355] - Handle accidental deletion of source-topics as exceptional failure
[KAFKA-10367] - Allow running the Streams demo app with a config file
[KAFKA-10403] - Replace scala collection by java collection in generating MBeans attributes
[KAFKA-10407] - add linger.ms parameter support to KafkaLog4jAppender
[KAFKA-10418] - Incomplete error/docs when altering topic configs via kafka-topics with --bootstrap-server
[KAFKA-10458] - Need a way to update quota for TokenBucket registered with Sensor
[KAFKA-10459] - Document IQ APIs where order does not hold between stores
[KAFKA-10478] - advertised.listeners should allow duplicated ports
[KAFKA-10494] - Streams: enableSendingOldValues should not call parent if node is itself materialized
[KAFKA-10503] - MockProducer doesn't throw ClassCastException when no partition for topic
[KAFKA-10509] - Add metric to track throttle time due to hitting connection rate quota
[KAFKA-10521] - Remove ZK watch for completing partition reassignment
[KAFKA-10584] - IndexSearchType should use sealed trait instead of Enumeration
[KAFKA-10611] - Merge log error to avoid double error
[KAFKA-10801] - Docs on configuration have multiple places using the same HTML anchor tag
[KAFKA-10802] - Spurious log message when starting consumers

Bug

[KAFKA-6078] - Investigate failure of ReassignPartitionsClusterTest.shouldExpandCluster
[KAFKA-7970] - Missing topic causes service shutdown without exception
[KAFKA-8033] - Flaky Test PlaintextConsumerTest#testFetchInvalidOffset
[KAFKA-8098] - Flaky Test AdminClientIntegrationTest#testConsumerGroups
[KAFKA-8264] - Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[KAFKA-8334] - Occasional OffsetCommit Timeout
[KAFKA-8362] - LogCleaner gets stuck after partition move between log directories
[KAFKA-8630] - Unit testing a streams processor with a WindowStore throws a ClassCastException
[KAFKA-9066] - Kafka Connect JMX : source & sink task metrics missing for tasks in failed state
[KAFKA-9381] - Javadocs + Scaladocs not published on maven central
[KAFKA-9514] - The protocol generator generated useless condition when a field is made nullable and flexible version is used
[KAFKA-9516] - Flaky Test PlaintextProducerSendTest#testNonBlockingProducer
[KAFKA-9584] - Removing headers causes ConcurrentModificationException
[KAFKA-9666] - Transactional producer Epoch could not be reset
[KAFKA-9769] - ReplicaManager Partition.makeFollower Increases LeaderEpoch when ZooKeeper disconnect occurs
[KAFKA-9845] - plugin.path property does not work with config provider
[KAFKA-10014] - Always try to close all channels in Selector#close
[KAFKA-10048] - Possible data gap for a consumer after a failover when using MM2
[KAFKA-10079] - Improve thread-level stickiness of stateful tasks
[KAFKA-10109] - kafka-acls.sh/AclCommand opens multiple AdminClients
[KAFKA-10120] - DescribeLogDirsResult exposes internal classes
[KAFKA-10133] - Cannot compress messages in destination cluster with MM2
[KAFKA-10134] - High CPU issue during rebalance in Kafka consumer after upgrading to 2.5
[KAFKA-10140] - Incremental config api excludes plugin config changes
[KAFKA-10147] - MockAdminClient#describeConfigs(Collection) is unable to handle broker resource
[KAFKA-10158] - Fix flaky kafka.admin.TopicCommandWithAdminClientTest#testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress
[KAFKA-10160] - Kafka MM2 consumer configuration
[KAFKA-10179] - State Store Passes Wrong Changelog Topic to Serde for Optimized Source Tables
[KAFKA-10180] - TLSv1.3 system tests should not run under Java 8
[KAFKA-10188] - Sink task preCommit method gets called after task is stopped
[KAFKA-10189] - Reset metric EventQueueTimeMs
[KAFKA-10192] - Flaky test BlockingConnectorTest#testBlockInConnectorStop
[KAFKA-10193] - Add preemption for controller events that have callbacks
[KAFKA-10200] - MockProcessorContext doesn't work with WindowStores
[KAFKA-10215] - MockProcessorContext doesn't work with SessionStores
[KAFKA-10218] - DistributedHerder's canReadConfigs field is never reset to true
[KAFKA-10220] - NPE when describing resources
[KAFKA-10224] - The license term about jersey is not correct
[KAFKA-10235] - Fix flaky transactions_test.py
[KAFKA-10240] - Sink tasks should not throw WakeupException on shutdown
[KAFKA-10243] - ConcurrentModificationException while processing connection setup timeouts
[KAFKA-10268] - dynamic config like "--delete-config log.retention.ms" doesn't work
[KAFKA-10271] - Performance regression while fetching a key from a single partition
[KAFKA-10272] - kafka-server-stop.sh fails on IBM i
[KAFKA-10282] - Log metrics are removed if a log is deleted and re-created quickly enough
[KAFKA-10284] - Group membership update due to static member rejoin should be persisted
[KAFKA-10300] - fix flaky core/group_mode_transactions_test.py
[KAFKA-10312] - MetadataCache.getPartitionMetadata may return incorrect error code when partition's leader is not present at the MetadataCache
[KAFKA-10314] - KafkaStorageException on reassignment when offline log directories exist
[KAFKA-10321] - shouldDieOnInvalidOffsetExceptionDuringStartup would block forever on JDK11
[KAFKA-10326] - Both serializer and deserializer should be able to see the generated client id
[KAFKA-10332] - MirrorMaker2 fails to detect topic if remote topic is created first
[KAFKA-10340] - Source connectors should report error when trying to produce records to non-existent topics instead of hanging forever
[KAFKA-10362] - When resuming Streams active task with EOS, the checkpoint file should be deleted
[KAFKA-10384] - Separate converters from generated messages
[KAFKA-10387] - Cannot include SMT configs with source connector that uses topic.creation.* properties
[KAFKA-10391] - Streams should overwrite checkpoint excluding corrupted partitions
[KAFKA-10395] - TopologyTestDriver does not work with dynamic topic routing
[KAFKA-10401] - GroupMetadataManager ignores current_state_timestamp field for GROUP_METADATA_VALUE_SCHEMA_V3
[KAFKA-10426] - Deadlock in KafkaConfigBackingStore
[KAFKA-10439] - Connect's Values class loses precision for integers, larger than 64 bits
[KAFKA-10454] - Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[KAFKA-10455] - Probing rebalances are not guaranteed to be triggered by non-leader members
[KAFKA-10463] - the necessary utilities in Dockerfile should include git
[KAFKA-10477] - Sink Connector fails with DataException when trying to convert Kafka record with empty key to Connect Record
[KAFKA-10479] - Throw exception if users try to update configs of existent listeners
[KAFKA-10502] - Threadlocal may can not set null,because it may create a memory leak
[KAFKA-10515] - NPE: Foreign key join serde may not be initialized with default serde if application is distributed
[KAFKA-10520] - InitProducerId may be blocked if least loaded node is not ready to send
[KAFKA-10531] - KafkaBasedLog can sleep for negative values
[KAFKA-10557] - Missing docs when describing topic configs including documentation
[KAFKA-10559] - Don't shutdown the entire app upon TimeoutException during internal topic validation
[KAFKA-10564] - Continuous logging about deleting obsolete state directories
[KAFKA-10574] - Infinite loop in SimpleHeaderConverter and Values classes
[KAFKA-10598] - KafkaStreams reports inappropriate error message for IQ
[KAFKA-10600] - Connect adds error to property in validation result if connector does not define the property
[KAFKA-10602] - DLQ Reporter throws NPE when reporting from different thread
[KAFKA-10606] - Auto create non-existent topics when fetching metadata for all topics
[KAFKA-10613] - Broker should not set leader epoch if the list-offset request version < 4
[KAFKA-10616] - StreamThread killed by "IllegalStateException: The processor is already closed"
[KAFKA-10631] - ProducerFencedException is not Handled on Offest Commit
[KAFKA-10638] - QueryableStateIntegrationTest fails due to stricter store checking
[KAFKA-10647] - Only serialize owned partition when consumer protocol version >= 0
[KAFKA-10651] - Assignor reports offsets from uninitialized task
[KAFKA-10664] - Streams fails to overwrite corrupted offsets leading to infinite OffsetOutOfRangeException loop
[KAFKA-10687] - Produce request should be bumped for new error code PRODUCE_FENCED
[KAFKA-10755] - Should consider commit latency when computing next commit timestamp
[KAFKA-10758] - Kafka Streams consuming from a pattern goes to PENDING_SHUTDOWN when adding a new topic
[KAFKA-10799] - AlterIsr path does not update ISR shrink/expand meters
[KAFKA-10811] - System exit from MirrorConnectorsIntegrationTest#testReplication
[KAFKA-10813] - StreamsProducer should catch InvalidProducerEpoch and throw TaskMigrated in all cases

Task

[KAFKA-10068] - Verify HighAvailabilityTaskAssignor performance with large clusters and topologies
[KAFKA-10084] - System test failure: StreamsEosTest.test_failure_and_recovery_complex
[KAFKA-10153] - Error Reporting in Connect Documentation
[KAFKA-10341] - Add version 2.6 to streams and systems tests
[KAFKA-10456] - wrong description in kafka-console-producer.sh help
[KAFKA-10556] - NPE if sasl.mechanism is unrecognized
[KAFKA-10592] - system tests not running after python3 merge
[KAFKA-10669] - ListOffsetRequest: make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field to 1

Test

[KAFKA-9273] - Refactor AbstractJoinIntegrationTest and Sub-classes
[KAFKA-9943] - Enable TLSv.1.3 in system tests "run all" execution.
[KAFKA-10191] - fix flaky StreamsOptimizedTest
[KAFKA-10255] - Fix flaky testOneWayReplicationWithAutorOffsetSync1 test
[KAFKA-10286] - Connect system tests should wait for workers to join group
[KAFKA-10295] - ConnectDistributedTest.test_bounce should wait for graceful stop
[KAFKA-10351] - Add missing tests for IOExceptions for GlobalStateManagerImpl
[KAFKA-10404] - Flaky Test kafka.api.SaslSslConsumerTest.testCoordinatorFailover

Sub-task

[KAFKA-9432] - Replace DescribeConfigs request/response with automated protocol
[KAFKA-9434] - Replace AlterReplicaLogDirs request/response with automated protocol
[KAFKA-9627] - Replace ListOffset request/response with automated protocol
[KAFKA-9629] - Replace Fetch request/response with automated protocol
[KAFKA-9911] - Implement new producer fenced error
[KAFKA-10027] - Implement read path for feature versioning scheme
[KAFKA-10028] - Implement write path for feature versioning scheme
[KAFKA-10042] - Make INVALID_PRODUCER_EPOCH abortable from Produce response
[KAFKA-10163] - Implement Broker side changes
[KAFKA-10164] - Implement Admin side changes
[KAFKA-10174] - Prefer --bootstrap-server ducktape tests using kafka_configs.sh
[KAFKA-10213] - Prefer --bootstrap-server in ducktape tests for Kafka clients
[KAFKA-10258] - Get rid of use_zk_connection flag in kafka.py public methods
[KAFKA-10261] - Introduce the KIP-478 processors with shims
[KAFKA-10270] - Add a broker to controller channel manager to redirect AlterConfig
[KAFKA-10308] - fix flaky core/round_trip_fault_test.py
[KAFKA-10319] - Fix unknown offset sum on trunk
[KAFKA-10379] - Implement the KIP-478 StreamBuilder#addGlobalStore()
[KAFKA-10435] - Fetch protocol changes for KIP-595
[KAFKA-10437] - KIP-478: Implement test-utils changes
[KAFKA-10447] - Migrate tools module to JUnit 5
[KAFKA-10516] - Implement Topic Command changes
[KAFKA-10535] - KIP-478: Implement StateStoreContext and Record
[KAFKA-10536] - KIP-478: Implement KStream changes
[KAFKA-10562] - KIP-478: Delegate the store wrappers to the new init method
[KAFKA-10570] - Rename JMXReporter configs for KIP-629
[KAFKA-10571] - Replace occurrences of blackout with backoff for KIP-629
[KAFKA-10572] - Rename MirrorMaker 2 blacklist configs for KIP-629
[KAFKA-10573] - Rename connect transform configs for KIP-629
[KAFKA-10599] - Implement basic CLI tool for feature versioning system
[KAFKA-10605] - KIP-478: deprecate the replaced Processor API members