Release Notes - Kafka - Version 2.4.0

Below is a summary of the JIRA issues addressed in the 2.4.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

• [KAFKA-3333] - Alternative Partitioner to Support "Always Round-Robin" partitioning
• [KAFKA-3705] - Support non-key joining in KTable
• [KAFKA-7471] - Multiple Consumer Group Management (Describe, Reset, Delete)
• [KAFKA-7500] - MirrorMaker 2.0 (KIP-382)
• [KAFKA-7800] - Extend Admin API to support dynamic application log levels
• [KAFKA-8286] - KIP-460 Admin Leader Election RPC
• [KAFKA-8447] - New Metric to Measure Number of Tasks on a Connector
• [KAFKA-8874] - KIP-517: Add consumer metrics to observe user poll behavior
• [KAFKA-8885] - The Kafka Protocol should Support Optional Tagged Fields
• [KAFKA-8907] - Return topic configs in CreateTopics response
• [KAFKA-8952] - Vulnerabilities found for jackson-databind-2.9.9

Improvement

• [KAFKA-5609] - Connect log4j should log to file by default
• [KAFKA-6263] - Expose metric for group metadata loading duration
• [KAFKA-6883] - KafkaShortnamer should allow to convert Kerberos principal name to upper case user name
• [KAFKA-6958] - Allow to define custom processor names with KStreams DSL
• [KAFKA-7018] - persist memberId for consumer restart
• [KAFKA-7149] - Reduce assignment data size to improve kafka streams scalability
• [KAFKA-7190] - Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
• [KAFKA-7197] - Support Scala 2.13.0
• [KAFKA-7245] - Deprecate WindowStore#put(key, value)
• [KAFKA-7273] - Converters should have access to headers.
• [KAFKA-7335] - Store clusterId locally to ensure broker joins the right cluster
• [KAFKA-7548] - KafkaConsumer should not throw away already fetched data for paused partitions.
• [KAFKA-7772] - Dynamically adjust log level in Connect workers
• [KAFKA-7981] - Add Replica Fetcher and Log Cleaner Count Metrics
• [KAFKA-8233] - KIP-470: TopologyTestDriver test input and output usability improvements
• [KAFKA-8305] - AdminClient should support creating topics with default partitions and replication factor
• [KAFKA-8333] - Load high watermark checkpoint only once when handling LeaderAndIsr requests
• [KAFKA-8400] - Do not update follower replica state if the log read failed
• [KAFKA-8452] - Possible Suppress buffer optimization: de-duplicate prior value
• [KAFKA-8488] - FetchSessionHandler logging create 73 mb allocation in TLAB which could be no op
• [KAFKA-8501] - Remove key and value from exception message
• [KAFKA-8513] - Add kafka-streams-application-reset.bat for Windows platform
• [KAFKA-8544] - Remove legacy kafka.admin.AdminClient
• [KAFKA-8545] - Remove legacy ZkUtils
• [KAFKA-8558] - KIP-479 - Add StreamJoined Overload to KStream#Join
• [KAFKA-8559] - PartitionStates.partitionStates cause array grow allocation.
• [KAFKA-8560] - The Kafka protocol generator should support common structures
• [KAFKA-8563] - Minor: Remove method call in networkSend. Rely on java's vargs boxing/autoboxing
• [KAFKA-8594] - Add Kafka Streams compatibility test for Kafka 2.3.0
• [KAFKA-8595] - Support SerDe of Decimals in JSON that are not HEX encoded
• [KAFKA-8601] - Producer Improvement: Sticky Partitioner
• [KAFKA-8634] - Update ZooKeeper to 3.5.5
• [KAFKA-8644] - The Kafka protocol generator should allow null defaults for bytes and array fields
• [KAFKA-8669] - Add java security providers in Kafka Security config
• [KAFKA-8676] - Avoid Stopping Unnecessary Connectors and Tasks
• [KAFKA-8696] - Clean up Sum/Count/Total metrics
• [KAFKA-8717] - Use cached hw/lso offset metadata when reading from log
• [KAFKA-8725] - Improve LogCleaner error handling when failing to grab the filthiest log
• [KAFKA-8730] - Add API to delete consumer offsets (KIP-496)
• [KAFKA-8745] - DumpLogSegments doesn't show keys, when the message is null
• [KAFKA-8753] - Add JMX for number of topics marked for deletion
• [KAFKA-8792] - Default ZK configuration to disable AdminServer
• [KAFKA-8839] - Improve logging in Kafka Streams around debugging task lifecycle
• [KAFKA-8853] - Create sustained connections test for Trogdor
• [KAFKA-8859] - Refactor Cache-level Streams Metrics
• [KAFKA-8880] - Augment Consumer.committed(partition) to allow multiple partitions
• [KAFKA-8884] - Improve streams errors on class cast exception in ProcessorsNodes
• [KAFKA-8927] - Remove config `partition.grouper` and interface `PartitionGrouper`
• [KAFKA-8934] - Introduce Instance-level Metrics
• [KAFKA-8954] - Topic existence check is wrongly implemented in the DeleteOffset API
• [KAFKA-8956] - Refactor DelayedCreatePartitions#updateWaiting to avoid modifying collection in foreach
• [KAFKA-8984] - Improve tagged fields documentation
• [KAFKA-8985] - Use flexibleVersions with LeaderAndIsr, UMR, etc., and improve RequestResponseTest coverage
• [KAFKA-9020] - Streams sub-topologies should be sorted by sink -> source relationship
• [KAFKA-9033] - Change default client-id in consumer / producer to be more meaningful
• [KAFKA-9038] - Allow creating partitions while partition reassignment is in progress
• [KAFKA-9058] - Foreign Key Join should not require a queriable store

Bug

• [KAFKA-1714] - more better bootstrapping of the gradle-wrapper.jar
• [KAFKA-5566] - Flaky Test QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied
• [KAFKA-5998] - /.checkpoint.tmp Not found exception
• [KAFKA-6290] - Kafka Connect cast transformation should support logical types
• [KAFKA-6605] - Flatten SMT does not properly handle fields that are null
• [KAFKA-7157] - Connect TimestampConverter SMT doesn't handle null values
• [KAFKA-7263] - Container exception java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null
• [KAFKA-7849] - Warning when adding GlobalKTable
• [KAFKA-7921] - Instable KafkaStreamsTest
• [KAFKA-7937] - Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
• [KAFKA-7941] - Connect KafkaBasedLog work thread terminates when getting offsets fails because broker is unavailable
• [KAFKA-7990] - Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs
• [KAFKA-7994] - Improve Stream-Time for rebalances and restarts
• [KAFKA-8024] - UtilsTest.testFormatBytes fails with german locale
• [KAFKA-8041] - Flaky Test LogDirFailureTest#testIOExceptionDuringLogRoll
• [KAFKA-8053] - kafka-topics.sh gives confusing error message when the topic doesn't exist
• [KAFKA-8078] - Flaky Test TableTableJoinIntegrationTest#testInnerInner
• [KAFKA-8086] - Flaky Test GroupAuthorizerIntegrationTest#testPatternSubscriptionWithTopicAndGroupRead
• [KAFKA-8104] - Consumer cannot rejoin to the group after rebalancing
• [KAFKA-8106] - Reducing the allocation and copying of ByteBuffer when logValidator do validation.
• [KAFKA-8122] - Flaky Test EosIntegrationTest#shouldNotViolateEosIfOneTaskFailsWithState
• [KAFKA-8198] - KStreams testing docs use non-existent method "pipe"
• [KAFKA-8262] - Flaky Test MetricsIntegrationTest#testStreamMetric
• [KAFKA-8263] - Flaky Test MetricsIntegrationTest#testStreamMetricOfWindowStore
• [KAFKA-8297] - Kafka Streams ConsumerRecordFactory yields difficult compiler error about generics
• [KAFKA-8319] - Flaky Test KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory
• [KAFKA-8340] - ServiceLoader fails when used from isolated plugin path directory
• [KAFKA-8392] - Kafka broker leaks metric when partition leader moves to another node.
• [KAFKA-8412] - Still a nullpointer exception thrown on shutdown while flushing before closing producers
• [KAFKA-8427] - Error while cleanup under windows for EmbeddedKafkaCluster
• [KAFKA-8442] - Inconsistent ISR output in topic command when using --bootstrap-server
• [KAFKA-8448] - Too many kafka.log.Log instances (Memory Leak)
• [KAFKA-8454] - Add Java AdminClient interface
• [KAFKA-8456] - Flaky Test StoreUpgradeIntegrationTest#shouldMigratePersistentWindowStoreToTimestampedWindowStoreUsingPapi
• [KAFKA-8461] - Flakey test UncleanLeaderElectionTest#testUncleanLeaderElectionDisabledByTopicOverride
• [KAFKA-8487] - Consumer should not resetGeneration upon REBALANCE_IN_PROGRESS in commit response handler
• [KAFKA-8514] - Kafka clients should not include Scala's Java 8 compatibility lib
• [KAFKA-8523] - InsertField transformation fails when encountering tombstone event
• [KAFKA-8526] - Broker may select a failed dir for new replica even in the presence of other live dirs
• [KAFKA-8530] - Consumer should handle authorization errors in OffsetFetch
• [KAFKA-8550] - Connector validation fails with aliased converters
• [KAFKA-8586] - Source task producers silently fail to send records
• [KAFKA-8602] - StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic
• [KAFKA-8615] - Change to track partition time breaks TimestampExtractor
• [KAFKA-8620] - Race condition in StreamThread state change
• [KAFKA-8635] - Unnecessary wait when looking up coordinator before transactional request
• [KAFKA-8637] - WriteBatch objects leak off-heap memory
• [KAFKA-8649] - Error while rolling update from Kafka Streams 2.0.0 -> Kafka Streams 2.1.0
• [KAFKA-8662] - Produce fails if a previous produce was to an unauthorized topic
• [KAFKA-8670] - kafka-topics.sh shows IllegalArgumentException when describing all topics if no topics exist on the cluster
• [KAFKA-8671] - NullPointerException occurs if topic associated with GlobalKTable changes
• [KAFKA-8677] - Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
• [KAFKA-8688] - Upgrade system tests fail due to data loss with older message format
• [KAFKA-8698] - ListOffsets Response protocol documentation
• [KAFKA-8731] - InMemorySessionStore throws NullPointerException on startup
• [KAFKA-8736] - Performance: ThreadCache uses size() for empty cache check
• [KAFKA-8742] - Flaky Test RepartitionWithMergeOptimizingIntegrationTest#shouldSendCorrectResults_NO_OPTIMIZATION
• [KAFKA-8743] - Flaky Test RepartitionWithMergeOptimizingIntegrationTest#shouldSendCorrectRecords_OPTIMIZED
• [KAFKA-8748] - Flaky Test DescribeLogDirsRequestTest#testDescribeLogDirsRequest
• [KAFKA-8755] - Stand-by Task of an Optimized Source Table Does Not Write Anything to its State Store
• [KAFKA-8774] - Connect REST API exposes plaintext secrets in tasks endpoint if config value contains additional characters
• [KAFKA-8782] - ReplicationQuotaManagerTest and ClientQuotaManagerTest should close Metrics object
• [KAFKA-8788] - Optimize client metadata handling with a large number of partitions
• [KAFKA-8791] - RocksDBTimestampedStore should open in regular mode for new stores
• [KAFKA-8801] - electLeaderForPartitions redundancy for some success elect partitions
• [KAFKA-8802] - ConcurrentSkipListMap shows performance regression in cache and in-memory store
• [KAFKA-8804] - Internal Connect REST endpoints are insecure
• [KAFKA-8813] - Race condition when creating topics and changing their configuration
• [KAFKA-8816] - RecordCollector offsets updated indirectly by StreamTask
• [KAFKA-8817] - Flaky Test KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest
• [KAFKA-8819] - Plugin path for converters not working as intended
• [KAFKA-8824] - InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured
• [KAFKA-8837] - Several integration tests incorrectly use verifyNonDaemonThreadsStatus
• [KAFKA-8857] - Config describe should not return isReadOnly=false based on synonyms
• [KAFKA-8860] - SslPrincipalMapper should handle distinguished names with spaces
• [KAFKA-8861] - Fix flaky RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
• [KAFKA-8875] - CreateTopic API should check topic existence before replication factor
• [KAFKA-8878] - Flaky Test AssignedStreamsTasksTest#shouldCloseCleanlyWithSuspendedTaskAndEOS
• [KAFKA-8889] - Root cause is lost for FetchSessionHandler.handleError
• [KAFKA-8891] - invalid assignment protocol java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null
• [KAFKA-8892] - Display the sorted configs in Kafka Configs Help Command.
• [KAFKA-8911] - Implicit TimeWindowedSerde creates Serde with null inner serializer
• [KAFKA-8943] - Move SecurityProviderCreator to a public package
• [KAFKA-8945] - Incorrect null check in the constructor for ConnectorHealth and AbstractState
• [KAFKA-8947] - Connect framework incorrectly instantiates TaskStates for REST extensions
• [KAFKA-8950] - KafkaConsumer stops fetching
• [KAFKA-8962] - KafkaAdminClient#describeTopics always goes through the controller
• [KAFKA-8972] - KafkaConsumer.unsubscribe could leave inconsistent user rebalance callback state
• [KAFKA-8974] - Sink Connectors can't handle topic list with whitespaces
• [KAFKA-8983] - AdminClient deleteRecords should not fail all partitions unnecessarily
• [KAFKA-8992] - Don't expose Errors in `RemoveMemberFromGroupResult`
• [KAFKA-9003] - Flaky Test RepartitionOptimizingIntegrationTest#shouldSendCorrectRecords_OPTIMIZED
• [KAFKA-9014] - AssertionError thrown by SourceRecordWriteCounter when SourceTask.poll returns an empty list
• [KAFKA-9029] - Flaky Test CooperativeStickyAssignorTest.testReassignmentWithRandomSubscriptionsAndChanges
• [KAFKA-9030] - Document Instance-level Metrics
• [KAFKA-9032] - Serializer is invoked with null when foreignValue=null
• [KAFKA-9046] - Connect worker configs require undocumented 'admin.' prefix to configure DLQ for connectors
• [KAFKA-9053] - AssignmentInfo#encode hardcodes the LATEST_SUPPORTED_VERSION
• [KAFKA-9069] - Flaky Test AdminClientIntegrationTest.testCreatePartitions
• [KAFKA-9073] - Kafka Streams State stuck in rebalancing after one of the StreamThread encounters java.lang.IllegalStateException: No current assignment for partition
• [KAFKA-9078] - System Test Failure: ConnectRestApiTest
• [KAFKA-9079] - System Test Failure: TransactionsTest
• [KAFKA-9080] - System Test Failure: MessageFormatChangeTest.testCompatibilty
• [KAFKA-9084] - AlterPartitionReassignments should not allow all replicas to be removed
• [KAFKA-9089] - Reassignment should be resilient to unexpected errors
• [KAFKA-9133] - LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
• [KAFKA-9140] - Consumer gets stuck rejoining the group indefinitely
• [KAFKA-9150] - DescribeGroup uses member assignment as metadata
• [KAFKA-9156] - LazyTimeIndex & LazyOffsetIndex may cause niobufferoverflow in concurrent state
• [KAFKA-9171] - DelayedFetch completion may throw exception, causing successful produce to be failed
• [KAFKA-9175] - MirrorMaker 2 emits invalid topic partition metrics
• [KAFKA-9178] - restoredPartitions is not cleared until the last restoring task completes
• [KAFKA-9184] - Redundant task creation and periodic rebalances after zombie worker rejoins the group
• [KAFKA-9190] - Server leaves connections with expired authentication sessions open
• [KAFKA-9196] - Records exposed before advancement of high watermark after segment roll
• [KAFKA-9198] - StopReplica handler should complete pending purgatory operations
• [KAFKA-9200] - ListOffsetRequest missing error response for v5
• [KAFKA-9203] - kafka-client 2.3.1 fails to consume lz4 compressed topic
• [KAFKA-9212] - Keep receiving FENCED_LEADER_EPOCH while sending ListOffsetRequest
• [KAFKA-9219] - NullPointerException when polling metrics from Kafka Connect
• [KAFKA-9231] - Streams Threads may die from recoverable errors with EOS enabled
• [KAFKA-9244] - Update of old FK reference on RHS should not trigger join result
• [KAFKA-9251] - Describing a non consumer group with the Admin API hangs forever
• [KAFKA-9258] - Connect ConnectorStatusMetricsGroup sometimes NPE
• [KAFKA-9261] - NPE when updating client metadata
• [KAFKA-9265] - kafka.log.Log instances are leaking on log delete

Task

• [KAFKA-8443] - Allow broker to select a preferred read replica for consumer
• [KAFKA-8765] - Remove "unstable" annotations
• [KAFKA-9072] - Add Section to Streams Developer Guide for Topology Naming (KIP-307)

Test

• [KAFKA-9000] - Flaky Test KTableKTableForeignKeyJoinIntegrationTest.doLeftJoinFromRightThenDeleteRightEntity
• [KAFKA-9123] - Add system test with large number of partitions

Sub-task

• [KAFKA-4222] - Flaky Test QueryableStateIntegrationTest.queryOnRebalance
• [KAFKA-4600] - Consumer proceeds on when ConsumerRebalanceListener fails
• [KAFKA-8221] - Augment LeaveGroupRequest to batch operation
• [KAFKA-8331] - Add system test for enabling static membership on KStream
• [KAFKA-8355] - Add static membership to Range assignor
• [KAFKA-8390] - Replace CreateDelegationToken request/response with automated protocol
• [KAFKA-8471] - Replace control requests/responses with automated protocol
• [KAFKA-8492] - Modify ConsumerCoordinator Algorithm with incremental protocol (part 2)
• [KAFKA-8493] - Add PartitionsLost API in RebalanceListener (part 3)
• [KAFKA-8496] - Add system test for compatibility and upgrade path (part 6)
• [KAFKA-8510] - Update StreamsPartitionAssignor to use the built-in owned partitions to achieve stickiness (part 7)
• [KAFKA-8539] - Add `group.instance.id` to Subscription class
• [KAFKA-8578] - Add Functionality to Expose RocksDB Metrics
• [KAFKA-8579] - Expose RocksDB Metrics to JMX
• [KAFKA-8580] - Compute RocksDB Metrics
• [KAFKA-8598] - Replace RenewDelegationToken request/response with automated protocol
• [KAFKA-8599] - Replace ExpireDelegationToken request/response with automated protocol
• [KAFKA-8600] - Replace DescribeDelegationToken request/response with automated protocol
• [KAFKA-8603] - Document upgrade path
• [KAFKA-8609] - Add consumer metrics for rebalances (part 9)
• [KAFKA-8704] - Add PartitionAssignor adapter for backwards compatibility
• [KAFKA-8760] - KIP-504: Add new Java Authorizer API
• [KAFKA-8848] - Update system test to use new authorizer
• [KAFKA-8866] - Make Authorizer create/delete exceptions Optional
• [KAFKA-8886] - Make Authorizer create/delete methods asynchronous
• [KAFKA-8887] - Use purgatory for CreateAcls and DeleteAcls if implementation is async
• [KAFKA-8902] - Benchmark cooperative vs eager rebalancing
• [KAFKA-8932] - Add tag for TopicConfigErrorCode in CreateTopics response when KIP-482 is merged
• [KAFKA-8942] - Document RocksDB metrics