Release Notes - Kafka - Version 3.5.0

Below is a summary of the JIRA issues addressed in the 3.5.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

Improvement

[KAFKA-6586] - Refactor Connect executables
[KAFKA-7109] - KafkaConsumer should close its incremental fetch sessions on close
[KAFKA-7499] - Extend ProductionExceptionHandler to cover serialization exceptions
[KAFKA-10244] - An new java interface to replace 'kafka.common.MessageReader'
[KAFKA-10575] - StateRestoreListener#onRestoreEnd should always be triggered
[KAFKA-12446] - Define KGroupedTable#aggregate subtractor + adder order of execution
[KAFKA-12634] - Should checkpoint after restore finished
[KAFKA-13659] - MM2 should read all offset syncs at start up
[KAFKA-13771] - Support to explicitly delete delegationTokens that have expired but have not been automatically cleaned up
[KAFKA-13817] - Schedule nextTimeToEmit to system time every time instead of just once
[KAFKA-13999] - Add ProducerCount metrics (KIP-847)
[KAFKA-14021] - MirrorMaker 2 should implement KIP-618 APIs
[KAFKA-14084] - Support SCRAM when using KRaft mode
[KAFKA-14253] - StreamsPartitionAssignor should print the member count in assignment logs
[KAFKA-14285] - Delete quota node in zookeeper when configs are empty
[KAFKA-14351] - Implement controller mutation quotas in KRaft
[KAFKA-14365] - Extract common logic from Fetcher
[KAFKA-14376] - Add ConfigProvider to make use of environment variables
[KAFKA-14395] - Add config to configure client supplier for KafkaStreams
[KAFKA-14491] - Introduce Versioned Key-Value Stores to Kafka Streams
[KAFKA-14565] - Interceptor Resource Leak
[KAFKA-14570] - Problem description missing closing parenthesis symbol
[KAFKA-14610] - Publish Mirror Maker 2 offset syncs in task commit method
[KAFKA-14617] - Replicas with stale broker epoch should not be allowed to join the ISR
[KAFKA-14638] - Documentation for transaction.timeout.ms should be more precise
[KAFKA-14666] - MM2 should translate consumer group offsets behind replication flow
[KAFKA-14680] - Gradle version upgrade 7 -->> 8
[KAFKA-14720] - KIP-906: Tools migration guidelines
[KAFKA-14722] - Make BooleanSerde public
[KAFKA-14732] - Use an exponential backoff retry mechanism while reconfiguring connector tasks
[KAFKA-14740] - Missing source tag on MirrorSource metrics
[KAFKA-14745] - MirrorSourceConnector keeps creating ReplicationPolicy instances
[KAFKA-14758] - Extract inner classes from Fetcher for reuse in refactoring
[KAFKA-14765] - Support SCRAM for brokers at bootstrap
[KAFKA-14770] - Allow dynamic keystore update for brokers if string representation of DN matches even if canonical DNs don't match
[KAFKA-14771] - Include current thread ids in ConcurrentModificationException message
[KAFKA-14775] - Support SCRAM for broker to controller authentication
[KAFKA-14776] - Update SCRAM integration tests to run with KRaft
[KAFKA-14795] - Provide message formatter for RemoteLogMetadata
[KAFKA-14814] - Skip restart of connectors when redundant resume request is made
[KAFKA-14827] - Support for StandardAuthorizer in Benchmark
[KAFKA-14829] - Consolidate reassignment logic in PartitionReassignmentReplicas
[KAFKA-14834] - Improved processor semantics for versioned stores
[KAFKA-14837] - The MirrorCheckPointConnector of MM2 will rebalance frequently, when the source cluster group is many more and changes frequently (but the list of configured synchronous group does not change)
[KAFKA-14838] - MM2 Worker/Connector/Task clients should specify client ID based on flow and role
[KAFKA-14842] - MirrorCheckpointTask can reduce the rpc calls of "listConsumerGroupOffsets(group)" of irrelevant groups at each poll
[KAFKA-14881] - Update UserScramCredentialRecord for SCRAM ZK to KRaft migration
[KAFKA-14883] - Broker state should be "observer" in KRaft quorum
[KAFKA-14887] - ZK session timeout can cause broker to shutdown
[KAFKA-14988] - Upgrade scalaCollectionCompact to v2.9 for CVE-2022-36944

Bug

[KAFKA-5756] - Synchronization issue on flush
[KAFKA-6793] - Unnecessary warning log message
[KAFKA-6891] - send.buffer.bytes should be allowed to set -1 in KafkaConnect
[KAFKA-8713] - [Connect] JsonConverter NULL Values are replaced by default values even in NULLABLE fields
[KAFKA-9087] - ReplicaAlterLogDirs stuck and restart fails with java.lang.IllegalStateException: Offset mismatch for the future replica
[KAFKA-9981] - Running a dedicated mm2 cluster with more than one nodes,When the configuration is updated the task is not aware and will lose the update operation.
[KAFKA-12468] - Initial offsets are copied from source to target cluster
[KAFKA-12558] - MM2 may not sync partition offsets correctly
[KAFKA-12639] - AbstractCoordinator ignores backoff timeout when joining the consumer group
[KAFKA-13891] - sync group failed with rebalanceInProgress error cause rebalance many rounds in coopeartive
[KAFKA-14016] - Revoke more partitions than expected in Cooperative rebalance
[KAFKA-14054] - Unexpected client shutdown as TimeoutException is thrown as IllegalStateException
[KAFKA-14072] - Crashed MirrorCheckpointConnector appears as running in REST API
[KAFKA-14128] - Kafka Streams terminates on topic check
[KAFKA-14139] - Replaced disk can lead to loss of committed data even with non-empty ISR
[KAFKA-14172] - bug: State stores lose state when tasks are reassigned under EOS wit…
[KAFKA-14295] - FetchMessageConversionsPerSec meter not recorded
[KAFKA-14311] - Connect Worker clean shutdown does not cleanly stop connectors/tasks
[KAFKA-14317] - ProduceRequest timeouts are logged as network exceptions
[KAFKA-14420] - MirrorMaker should not clear filtered configs on target topics
[KAFKA-14455] - Kafka Connect create and update REST APIs should surface failures while writing to the config topic
[KAFKA-14463] - ConnectorClientConfigOverridePolicy is not closed at worker shutdown
[KAFKA-14531] - KRaft controller time-based snapshots are too frequent
[KAFKA-14544] - The "is-future" should be removed from metrics tags after future log becomes current log
[KAFKA-14545] - MirrorCheckpointTask throws NullPointerException when group hasn't consumed from some partitions
[KAFKA-14564] - Upgrade Netty to 4.1.86.Final to fix CVEs
[KAFKA-14639] - Kafka CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
[KAFKA-14644] - Process should stop after failure in raft IO thread
[KAFKA-14645] - Plugin classloader not used when retrieving connector plugin config defs via REST API
[KAFKA-14649] - Failures instantiating Connect plugins hides other plugins from REST API, or crash worker
[KAFKA-14650] - IQv2 can throw ConcurrentModificationException when accessing Tasks
[KAFKA-14659] - source-record-write-[rate|total] metrics include filtered records
[KAFKA-14660] - Divide by zero security vulnerability (sonatype-2019-0422)
[KAFKA-14664] - Raft idle ratio is inaccurate
[KAFKA-14676] - Token endpoint URL used for OIDC cannot be set on the JAAS config
[KAFKA-14693] - KRaft Controller and ProcessExitingFaultHandler can deadlock shutdown
[KAFKA-14704] - Follower should truncate before incrementing high watermark
[KAFKA-14717] - KafkaStreams can' get running if the rebalance happens before StreamThread gets shutdown completely
[KAFKA-14727] - Connect EOS mode should periodically call task commit
[KAFKA-14729] - The kafakConsumer pollForFetches(timer) method takes up a lot of cpu due to the abnormal exit of the heartbeat thread
[KAFKA-14743] - MessageConversionsTimeMs for fetch request metric is not updated
[KAFKA-14744] - NPE while converting OffsetFetch from version < 8 to version >= 8
[KAFKA-14774] - the removed listeners should not be reconfigurable
[KAFKA-14781] - MM2 logs misleading error during topic ACL sync when broker does not have authorizer configured
[KAFKA-14792] - Race condition in LazyIndex.get()
[KAFKA-14794] - Unable to deserialize base64 JSON strings
[KAFKA-14797] - MM2 does not emit offset syncs when conservative translation logic exceeds positive max.offset.lag
[KAFKA-14799] - Source tasks fail if connector attempts to abort empty transaction
[KAFKA-14800] - Upgrade snappy-java Version to 1.1.9.1
[KAFKA-14801] - Encoded sensitive configs are not decoded before migration
[KAFKA-14804] - Connect docs fail to build with Gradle Swagger plugin 2.2.8
[KAFKA-14809] - Connect incorrectly logs that no records were produced by source tasks
[KAFKA-14812] - ProducerPerformance still counting successful sending in console when sending failed
[KAFKA-14816] - Connect loading SSL configs when contacting non-HTTPS URLs
[KAFKA-14836] - Fix UtilsTest#testToLogDateTimeFormat failure in some cases
[KAFKA-14839] - Exclude protected variable from JavaDocs
[KAFKA-14843] - Connector plugins config endpoint does not include Common configs
[KAFKA-14853] - the serializer/deserialize which extends ClusterResourceListener is not added to Metadata
[KAFKA-14862] - Outer stream-stream join does not output all results with multiple input partitions
[KAFKA-14864] - Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy
[KAFKA-14891] - Fix rack-aware range assignor to improve rack-awareness with co-partitioning
[KAFKA-14894] - MetadataLoader must call finishSnapshot after loading a snapshot
[KAFKA-14902] - KafkaBasedLog infinite retries can lead to StackOverflowError
[KAFKA-14943] - Fix ClientQuotaControlManager validation
[KAFKA-14946] - KRaft controller node shutting down while renouncing leadership
[KAFKA-14978] - ExactlyOnceWorkerSourceTask does not remove parent metrics
[KAFKA-14980] - MirrorMaker consumers don't get configs prefixed with source.cluster
[KAFKA-14994] - jose4j is vulnerable to CVE- Improper Cryptographic Algorithm
[KAFKA-14996] - The KRaft controller should properly handle overly large user operations
[KAFKA-15003] - TopicIdReplicaAssignment is not updated in migration (dual-write) when partitions are changed for topic
[KAFKA-15004] - Topic config changes are not synced during zk to kraft migration (dual-write)
[KAFKA-15007] - MV is not set correctly in the MetadataPropagator in migration.
[KAFKA-15009] - New ACLs are not written to ZK during migration
[KAFKA-15010] - KRaft Controller doesn't reconcile with Zookeeper metadata upon becoming new controller while in dual write mode.
[KAFKA-15015] - Binaries contain 2 versions of reload4j
[KAFKA-15019] - Improve handling of broker heartbeat timeouts
[KAFKA-15044] - Snappy v.1.1.9.1 NoClassDefFound on ARM machines

Task

[KAFKA-10586] - Full support for distributed mode in dedicated MirrorMaker 2.0 clusters
[KAFKA-14530] - Check state updater more than once in process loops
[KAFKA-14708] - Remove kafka.examples.Consumer dependancy on ShutdownableThread
[KAFKA-14731] - Upgrade ZooKeeper to 3.6.4
[KAFKA-14749] - Re-enable 'spotlessScalaCheck' task (in Jenkinsfile)
[KAFKA-14869] - txn and group coordinator downgrade foundation
[KAFKA-14974] - Restore backward compatibility in KafkaBasedLog
[KAFKA-14983] - Upgrade jetty-server to 9.4.51

Test

[KAFKA-12566] - Flaky Test MirrorConnectorsIntegrationSSLTest#testReplication
[KAFKA-14529] - fix failed KafkaServerKRaftRegistrationTest
[KAFKA-14534] - Flaky Test TransactionsExpirationTest.testTransactionAfterProducerIdExpires
[KAFKA-14905] - Failing tests in MM2 ForwardingAdmin test since KIP-894
[KAFKA-14929] - Flaky KafkaStatusBackingStoreFormatTest#putTopicStateRetriableFailure

Sub-task

[KAFKA-9550] - RemoteLogManager - copying eligible log segments to remote storage implementation
[KAFKA-9990] - Supporting transactions in tiered storage
[KAFKA-13020] - SnapshotReader should decode and report the append time in the header
[KAFKA-13369] - Follower fetch protocol enhancements for tiered storage.
[KAFKA-13884] - KRaft Obsever are not required to flush on every append
[KAFKA-14003] - Part 2
[KAFKA-14367] - Introduce `GroupCoordinator` interface
[KAFKA-14391] - Add ConsumerGroupHeartbeat API
[KAFKA-14425] - The Kafka protocol should support nullable structs
[KAFKA-14451] - Make range assignor rack-aware if consumer racks are configured
[KAFKA-14452] - Make sticky assignors rack-aware if consumer racks are configured.
[KAFKA-14466] - Refactor ClassLoaderAwareRemoteStorageManager.scala to ClassLoaderAwareRemoteStorageManager.java and move it to storage module.
[KAFKA-14468] - Refactor Commit Logic
[KAFKA-14471] - Move IndexEntry and related to storage module
[KAFKA-14472] - Move TransactionIndex and related to storage module
[KAFKA-14473] - Move AbstractIndex to storage module
[KAFKA-14474] - Move OffsetIndex to storage module
[KAFKA-14475] - Move TimeIndex/LazyIndex to storage module
[KAFKA-14476] - Move OffsetMap to storage module
[KAFKA-14477] - Move LogValidator and related to storage module
[KAFKA-14478] - Move LogConfig/CleanerConfig to storage module
[KAFKA-14480] - Move ProducerStateManager to storage module
[KAFKA-14495] - Improve the RemoteIndexCacheTest
[KAFKA-14521] - Replace BrokerCompressionCodec with BrokerCompressionType
[KAFKA-14543] - Move LogOffsetMetadata to storage module
[KAFKA-14549] - Move LogDirFailureChannel to storage module
[KAFKA-14550] - MoveSnapshotFile and CorruptSnapshotException to storage module
[KAFKA-14551] - Move LeaderEpochFileCache to storage module
[KAFKA-14554] - Move ClassLoaderAwareRemoteStorageManagerTest to storage module
[KAFKA-14558] - Move LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module.
[KAFKA-14568] - Move FetchDataInfo and related to storage module
[KAFKA-14575] - Move ClusterTool to tools
[KAFKA-14578] - Move ConsumerPerformance to tools
[KAFKA-14580] - Move EndToEndLatency to tools
[KAFKA-14582] - Move JmxTool to tools
[KAFKA-14586] - Move StreamsResetter to tools
[KAFKA-14590] - Move DelegationTokenCommand to tools
[KAFKA-14603] - Move KafkaMetricsGroup to server-common module.
[KAFKA-14607] - Move Scheduler/KafkaScheduler to server-common
[KAFKA-14611] - ZK broker should not send epoch during registration
[KAFKA-14628] - Move CommandLineUtils and CommandDefaultOptions shared classes
[KAFKA-14630] - Update zstd-jni version to 1.5.5
[KAFKA-14653] - MM2 should delay resolving config provider references
[KAFKA-14673] - Add high watermark listener to Partition/Log layers
[KAFKA-14678] - Move __consumer_offsets records from core to group-coordinator
[KAFKA-14679] - Add new __consumer_offsets records
[KAFKA-14685] - TierStateMachine interface for building remote aux log
[KAFKA-14688] - Move org.apache.kafka.server.log.internals to org.apache.kafka.storage.internals.log
[KAFKA-14706] - Move ShutdownableThread to server-commons module.
[KAFKA-14783] - Implement new STOPPED state for connectors
[KAFKA-14785] - Implement connector offset read REST API
[KAFKA-14793] - Propagate Topic Ids to the Group Coordinator during Offsets Commit
[KAFKA-14796] - Migrate ZK ACLs to KRaft
[KAFKA-14805] - KRaft Controller shouldn't allow metadata updates before migration starts
[KAFKA-14840] - Handle KRaft snapshots in dual-write mode
[KAFKA-14850] - Introduce InMemoryLeaderEpochCheckpoint to allow quick write/read
[KAFKA-14867] - Trigger rebalance when replica racks change if client.rack is configured
[KAFKA-14876] - Public documentation for new Kafka Connect offset management REST APIs in 3.5
[KAFKA-14909] - KRaft Controllers not setting ZkMigrationReady tagged field
[KAFKA-14918] - KRaft controller sending ZK controller RPCs to KRaft brokers