Release Notes - Kafka - Version 3.6.0

Below is a summary of the JIRA issues addressed in the 3.6.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

• [KAFKA-7739] - Kafka Tiered Storage
• [KAFKA-14305] - KRaft Metadata Transactions
• [KAFKA-14627] - Modernize Connect plugin discovery
• [KAFKA-15030] - Add connect-plugin-path command line tool
• [KAFKA-15031] - Add plugin.discovery worker configuration
• [KAFKA-15228] - Add sync-manifests subcommand to connect-plugin-path tool

Improvement

• [KAFKA-4107] - Support offset reset capability in Kafka Connect
• [KAFKA-8982] - Admin.deleteRecords should retry when failing to fetch metadata
• [KAFKA-12261] - Splitting partition causes message loss for consumers with auto.offset.reset=latest
• [KAFKA-13299] - Accept listeners that have the same port but use IPv4 vs IPv6
• [KAFKA-13431] - Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)
• [KAFKA-13504] - Retry connect internal topics' creation in case of InvalidReplicationFactorException
• [KAFKA-13875] - update docs to include topoicId for kafka-topics.sh --describe output
• [KAFKA-14038] - Optimize calculation of size for log in remote tier
• [KAFKA-14539] - Simplify StreamsMetadataState by replacing the Cluster metadata with partition info map
• [KAFKA-14661] - Upgrade Zookeeper to 3.8.2
• [KAFKA-14669] - Include MirrorMaker connector configurations in docs
• [KAFKA-14709] - Move content in connect/mirror/README.md to the docs
• [KAFKA-14735] - Improve KRaft metadata image change performance at high topic counts
• [KAFKA-14752] - improve kafka examples under examples package
• [KAFKA-14766] - Improve performance of VarInt encoding/decoding
• [KAFKA-14791] - Create a builder class for PartitionRegistration
• [KAFKA-14828] - Remove R/W lock from StandardAuthorizer
• [KAFKA-14866] - When broker shutdown, the controller module needs to remove its metrics
• [KAFKA-14868] - Remove some forgotten metrics when the replicaManager is closed
• [KAFKA-14926] - Remove metrics on Log Cleaner shutdown
• [KAFKA-14936] - Add Grace Period To Stream Table Join
• [KAFKA-14937] - Refactoring for client code to reduce boilerplate
• [KAFKA-14944] - Reduce CompletedFetch#parseRecord() memory copy
• [KAFKA-14982] - Improve the kafka-metadata-quorum output
• [KAFKA-14988] - Upgrade scalaCollectionCompact to v2.9 for CVE-2022-36944
• [KAFKA-14991] - Improving Producer's record timestamp validation
• [KAFKA-14993] - Improve TransactionIndex instance handling while copying to and fetching from RSM.
• [KAFKA-15034] - Improvement of ReplaceField performance for long list
• [KAFKA-15036] - Kraft leader change fails when invoking getFinalizedFeatures
• [KAFKA-15039] - Reduce logging level to trace in PartitionChangeBuilder.tryElection()
• [KAFKA-15076] - KRaft should prefer snapshots when listeners are at the start of the log
• [KAFKA-15078] - When fetching offset 0 the KRaft leader should response with SnapshotId
• [KAFKA-15085] - Make Timer.java implement AutoCloseable
• [KAFKA-15107] - Additional custom metadata for remote log segment
• [KAFKA-15121] - FileStreamSourceConnector and FileStreamSinkConnector should implement KIP-875 APIs (alterOffsets)
• [KAFKA-15123] - Add tests for ChunkedBytesStream
• [KAFKA-15126] - Change range queries to accept null lower and upper bounds
• [KAFKA-15130] - Delete remote segments when delete a topic
• [KAFKA-15131] - Improve RemoteStorageManager exception handling documentation
• [KAFKA-15139] - Optimize the performance of `Set.removeAll(List)` in `MirrorCheckpointConnector`
• [KAFKA-15141] - High CPU usage with log4j2
• [KAFKA-15153] - Use Python `is` instead of `==` to compare for None
• [KAFKA-15155] - Follow PEP 8 best practice in Python to check if a container is empty
• [KAFKA-15159] - Update minor dependencies in preparation for 3.5.1
• [KAFKA-15177] - MirrorMaker 2 should implement the alterOffsets KIP-875 API
• [KAFKA-15182] - Normalize offsets before invoking SourceConnector::alterOffsets
• [KAFKA-15183] - Add more controller, loader, snapshot emitter metrics
• [KAFKA-15213] - Provide the exact offset to QuorumController.replay
• [KAFKA-15219] - Support delegation tokens in KRaft
• [KAFKA-15222] - Upgrade zinc scala incremental compiler plugin version to a latests stable fit version (1.9.2)
• [KAFKA-15245] - Improve Tiered Storage Metrics
• [KAFKA-15291] - Implement Versioned interfaces in common Connect plugins
• [KAFKA-15336] - Connect plugin Javadocs should mention serviceloader manifests

Bug

• [KAFKA-8690] - Flakey test ConnectWorkerIntegrationTest#testAddAndRemoveWorke
• [KAFKA-9926] - Flaky test PlaintextAdminIntegrationTest.testCreatePartitions
• [KAFKA-10337] - Wait for pending async commits in commitSync() even if no offsets are specified
• [KAFKA-10579] - Flaky test connect.integration.InternalTopicsIntegrationTest.testStartWhenInternalTopicsCreatedManuallyWithCompactForBrokersDefaultCleanupPolicy
• [KAFKA-12525] - Inaccurate task status due to status record interleaving in fast rebalances in Connect
• [KAFKA-12842] - Failing test: org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest.testSourceTaskNotBlockedOnShutdownWithNonExistentTopic
• [KAFKA-13197] - KStream-GlobalKTable join semantics don't match documentation
• [KAFKA-13337] - Scanning for Connect plugins can fail with AccessDeniedException
• [KAFKA-13668] - Failed cluster authorization should not be fatal for producer
• [KAFKA-14273] - Kafka doesn't start with KRaft on Windows
• [KAFKA-14654] - Connectors have incorrect Thread Context Classloader during initialization
• [KAFKA-14662] - ACL listings in documentation are out of date
• [KAFKA-14694] - RPCProducerIdManager should not wait for a new block
• [KAFKA-14712] - Confusing error when writing downgraded FeatureImage
• [KAFKA-14831] - Illegal state errors should be fatal in transactional producer
• [KAFKA-14863] - Plugins which do not have a valid no-args constructor are visible in the REST API
• [KAFKA-14938] - Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary
• [KAFKA-14962] - Whitespace in ACL configuration causes Kafka startup to fail
• [KAFKA-14967] - MockAdminClient throws NullPointerException in CreateTopicsResult
• [KAFKA-14978] - ExactlyOnceWorkerSourceTask does not remove parent metrics
• [KAFKA-14997] - JmxToolTest failing with initializationError
• [KAFKA-15012] - JsonConverter fails when there are leading Zeros in a field
• [KAFKA-15016] - LICENSE-binary file contains dependencies not included anymore
• [KAFKA-15021] - KRaft controller increases leader epoch when shrinking ISR
• [KAFKA-15053] - Regression for security.protocol validation starting from 3.3.0
• [KAFKA-15059] - Exactly-once source tasks fail to start during pending rebalances
• [KAFKA-15077] - FileTokenRetriever doesn't trim the token before returning it.
• [KAFKA-15080] - Fetcher's lag never set when partition is idle
• [KAFKA-15091] - Javadocs for SourceTask::commit are incorrect
• [KAFKA-15096] - CVE 2023-34455 - Vulnerability identified with Apache kafka
• [KAFKA-15098] - KRaft migration does not proceed and broker dies if authorizer.class.name is set
• [KAFKA-15100] - Unsafe to call tryCompleteFetchResponse on request timeout
• [KAFKA-15102] - Mirror Maker 2 - KIP690 backward compatibility
• [KAFKA-15106] - AbstractStickyAssignor may stuck in 3.5
• [KAFKA-15109] - ISR shrink/expand issues on ZK brokers during migration
• [KAFKA-15114] - StorageTool help specifies user as parameter not name
• [KAFKA-15135] - RLM listener configurations passed but ignored by RLMM
• [KAFKA-15137] - Don't log the entire request in KRaftControllerChannelManager
• [KAFKA-15145] - AbstractWorkerSourceTask re-processes records filtered out by SMTs on retriable exceptions
• [KAFKA-15162] - Reflective plugin scanning misses plugins which are in parent classloaders but not classpath
• [KAFKA-15189] - Do not initialize RemoteStorage related metrics when disabled at cluster
• [KAFKA-15212] - Remove unneeded classgraph license file
• [KAFKA-15216] - InternalSinkRecord::newRecord method ignores the headers argument
• [KAFKA-15218] - NPE will be thrown while deleting topic and fetch from follower concurrently
• [KAFKA-15220] - KRaftMetadataCache returns fenced brokers from getAliveBrokerNode
• [KAFKA-15235] - No test coverage reports for Java due to settings for Jacoco being incompatible with Gradle 8.x
• [KAFKA-15238] - Connect workers can be disabled by DLQ-related blocking admin client calls
• [KAFKA-15243] - User creation mismatch
• [KAFKA-15244] - Connect PluginType.from(Class) result is incorrect when subclassing multiple plugin interfaces
• [KAFKA-15263] - KRaftMigrationDriver can run the migration twice
• [KAFKA-15312] - FileRawSnapshotWriter must flush before atomic move
• [KAFKA-15319] - Upgrade rocksdb to fix CVE-2022-37434
• [KAFKA-15338] - The metric group documentation for metrics added in KAFKA-13945 is incorrect
• [KAFKA-15345] - KRaft leader should notify the listener only when it has read up to the leader's epoch
• [KAFKA-15353] - Empty ISR returned from controller after AlterPartition request
• [KAFKA-15374] - ZK migration fails on configs for default broker resource
• [KAFKA-15375] - When running in KRaft mode, LogManager may creates CleanShutdown file by mistake
• [KAFKA-15377] - GET /connectors/{connector}/tasks-config endpoint exposes externalized secret values
• [KAFKA-15389] - MetadataLoader may publish an empty image on first start
• [KAFKA-15391] - Delete topic may lead to directory offline
• [KAFKA-15404] - Failing Test DynamicBrokerReconfigurationTest#testThreadPoolResize
• [KAFKA-15414] - remote logs get deleted after partition reassignment
• [KAFKA-15429] - Kafka Streams attempts to commit on a closed producer when shutting down after an exception when running with EOS
• [KAFKA-15435] - KRaft migration record counts in log message are incorrect
• [KAFKA-15441] - Broker sessions can time out during ZK migration
• [KAFKA-15450] - Disable ZK migration when JBOD configured
• [KAFKA-15473] - Connect connector-plugins endpoint shows duplicate plugins
• [KAFKA-15487] - CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1
• [KAFKA-15498] - Upgrade Snappy-Java to 1.1.10.4
• [KAFKA-15503] - CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1

Task

• [KAFKA-14559] - Handle object name with wildcards in the Jmx tool
• [KAFKA-14759] - Move test-only connectors from connect-runtime to test-specific module
• [KAFKA-14760] - Move ThroughputThrottler, break connect-runtime dependency on tools
• [KAFKA-14933] - Document Kafka Connect's log level REST APIs added in KIP-495
• [KAFKA-14950] - Implement assign() and assignment()
• [KAFKA-14966] - Extract reusable logic from OffsetFetcher
• [KAFKA-14974] - Restore backward compatibility in KafkaBasedLog
• [KAFKA-15069] - Refactor scanning hierarchy out of DelegatingClassLoader
• [KAFKA-15087] - Move InterBrokerSendThread to server-commons module
• [KAFKA-15150] - Add ServiceLoaderScanner implementation
• [KAFKA-15194] - Rename local tiered storage segment with offset as prefix for easy navigation
• [KAFKA-15233] - Add public documentation for plugin.discovery migration steps
• [KAFKA-15272] - Fix the logic which finds candidate log segments to upload it to tiered storage
• [KAFKA-15286] - Migrate ApiVersion related code to kraft
• [KAFKA-15400] - Fix flaky RemoteIndexCache test
• [KAFKA-15421] - Enable DynamicBrokerReconfigurationTest#testThreadPoolResize test
• [KAFKA-15422] - Update documentation for Delegation Tokens in Kafka with KRaft

Test

• [KAFKA-12384] - Flaky Test ListOffsetsRequestTest.testResponseIncludesLeaderEpoch
• [KAFKA-14682] - Unused stubbings are not reported by Mockito during CI builds
• [KAFKA-14718] - Flaky DedicatedMirrorIntegrationTest test suite
• [KAFKA-14905] - Failing tests in MM2 ForwardingAdmin test since KIP-894
• [KAFKA-15052] - Fix flaky test QuorumControllerTest.testBalancePartitionLeaders()
• [KAFKA-15148] - Some integration tests are running as unit tests
• [KAFKA-15180] - Generalize integration tests to change use of KafkaConsumer to Consumer
• [KAFKA-15211] - DistributedConfigTest#shouldFailWithInvalidKeySize fails when run after TestSslUtils#generate
• [KAFKA-15226] - System tests for plugin.discovery worker configuration
• [KAFKA-15239] - producerPerformance system test for old client failed after v3.5.0
• [KAFKA-15251] - Upgrade system test to use 3.5.1
• [KAFKA-15393] - MirrorMaker2 integration tests are shutting down uncleanly
• [KAFKA-15416] - Flaky test TopicAdminTest::retryEndOffsetsShouldRetryWhenTopicNotFound
• [KAFKA-15425] - Compatibility break in Admin.listOffsets() (2)
• [KAFKA-15439] - Add transaction tests enabled with tiered storage
• [KAFKA-15453] - Enable `testFencingOnTransactionExpiration` in TransactionsWithTieredStoreTest
• [KAFKA-15499] - Fix the flaky DeleteSegmentsDueToLogStartOffsetBreachTest

Sub-task

• [KAFKA-9564] - Integration Test framework for Tiered Storage
• [KAFKA-9579] - Remote consumer fetch implementation by adding respective purgatory
• [KAFKA-12969] - Add cluster or broker level config for topic level tiered storage confgs.
• [KAFKA-13187] - Replace EasyMock and PowerMock with Mockito for DistributedHerderTest
• [KAFKA-14059] - Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest
• [KAFKA-14278] - Convert INVALID_PRODUCER_EPOCH into PRODUCER_FENCED TxnOffsetCommit
• [KAFKA-14368] - Implement connector offset write REST API
• [KAFKA-14462] - New Group Coordinator State Machine
• [KAFKA-14500] - Implement JoinGroup/SyncGroup APIs
• [KAFKA-14501] - Implement Heartbeat API
• [KAFKA-14514] - Implement range broker side assignor
• [KAFKA-14518] - Rebalance on topic/partition metadata changes
• [KAFKA-14522] - Move RemoteIndexCache to the storage module
• [KAFKA-14561] - Improve transactions experience for older clients by ensuring ongoing transaction
• [KAFKA-14583] - Move ReplicaVerificationTool to tools
• [KAFKA-14584] - Deprecate StateChangeLogMerger tool
• [KAFKA-14591] - Move DeleteRecordsCommand to tools
• [KAFKA-14592] - Move FeatureCommand to tools
• [KAFKA-14594] - Move LogDirsCommand to tools
• [KAFKA-14632] - Compression optimization: Remove unnecessary intermediate buffers
• [KAFKA-14633] - Compression optimization: Use BufferSupplier to allocate the intermediate decompressed buffer
• [KAFKA-14647] - Move TopicFilter shared class
• [KAFKA-14702] - Extend server side assignor to support rack aware replica placement
• [KAFKA-14734] - Use CommandDefaultOptions in StreamsResetter
• [KAFKA-14737] - Move kafka.utils.json to server-common
• [KAFKA-14755] - improve java-producer-consumer-demo
• [KAFKA-14756] - improve exactly-once-demo example and ExactlyOnceMessageProcessor
• [KAFKA-14784] - Implement connector offset reset REST API
• [KAFKA-14851] - Move StreamResetterTest to tools
• [KAFKA-14884] - Include check transaction is still ongoing right before append
• [KAFKA-14888] - RemoteLogManager - deleting expired/size breached log segments to remote storage implementation
• [KAFKA-14920] - Address timeouts and out of order sequences
• [KAFKA-14930] - Public documentation for new Kafka Connect offset management REST APIs
• [KAFKA-14953] - Add metrics for tiered storage
• [KAFKA-15023] - Get rack information for source topic partitions for a task
• [KAFKA-15024] - Add cost function for task/client
• [KAFKA-15025] - Implement min-cost flow without balancing tasks for same subtopology
• [KAFKA-15027] - Implement rack aware assignment for standby tasks
• [KAFKA-15028] - AddPartitionsToTxnManager metrics
• [KAFKA-15037] - initialize unifiedLog with remoteStorageSystemEnable correctly
• [KAFKA-15040] - segment copy to remote storage won't work in KRaft mode
• [KAFKA-15054] - Add configs and logic to decide if rack aware assignment should be enabled
• [KAFKA-15066] - passing listener name config into TopicBasedRemoteLogMetadataManagerConfig
• [KAFKA-15083] - Passing "remote.log.metadata.*" configs into RLMM
• [KAFKA-15084] - Remove lock contention in RemoteIndexCache
• [KAFKA-15157] - Print startup time for RemoteIndexCache
• [KAFKA-15167] - Tiered Storage Test Harness Framework
• [KAFKA-15168] - Handle overlapping remote log segments in RemoteLogMetadata cache
• [KAFKA-15176] - Add missing tests for remote storage metrics
• [KAFKA-15181] - Race condition on partition assigned to TopicBasedRemoteLogMetadataManager
• [KAFKA-15199] - remove leading and trailing spaces from user input in release.py
• [KAFKA-15210] - Mention vote should be open for at atleast 72 hours
• [KAFKA-15232] - Move ToolsUtils to tools
• [KAFKA-15236] - Rename Remote Storage metrics to remove ambiguity
• [KAFKA-15246] - CoordinatorContext should be protected by a lock
• [KAFKA-15256] - Add code reviewers to contributors list in release email
• [KAFKA-15260] - RLM Task should wait until RLMM is initialized before copying segments to remote
• [KAFKA-15261] - ReplicaFetcher thread should not block if RLMM is not initialized
• [KAFKA-15267] - Cluster-wide disablement of Tiered Storage
• [KAFKA-15287] - Change NodeApiVersions.create() to contains both apis of zk and kraft broker
• [KAFKA-15288] - Change BrokerApiVersionsCommandTest to support kraft mode
• [KAFKA-15289] - Support KRaft mode in RequestQuotaTest
• [KAFKA-15290] - Add support to onboard existing topics to tiered storage
• [KAFKA-15293] - Update metrics doc to add tiered storage metrics
• [KAFKA-15294] - Make remote storage related configs as public (i.e. non-internal)
• [KAFKA-15295] - Add config validation when remote storage is enabled on a topic
• [KAFKA-15329] - Make default `remote.log.metadata.manager.class.name` as topic based RLMM
• [KAFKA-15351] - Update log-start-offset after leader election for topics enabled with remote storage
• [KAFKA-15352] - Ensure consistency while deleting the remote log segments
• [KAFKA-15380] - Try complete actions after callback
• [KAFKA-15399] - Enable OffloadAndConsumeFromLeader test
• [KAFKA-15410] - Add functional integration tests with tiered storage
• [KAFKA-15427] - Integration tests in TS test harness detect resource leaks
• [KAFKA-15442] - add document to introduce tiered storage feature and the usage
• [KAFKA-15459] - Convert coordinator retriable errors to a known producer response error.