Release Notes - Kafka - Version 2.8.0

Below is a summary of the JIRA issues addressed in the 2.8.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-10500] - Add API to Start and Stop Stream Threads
[KAFKA-10700] - Support mutual TLS authentication for SASL_SSL listeners
[KAFKA-10749] - Add throttling of IPs by connection rate

Improvement

[KAFKA-5488] - KStream.branch should not return a Array of streams we have to access by known index
[KAFKA-6181] - Examining log messages with {{--deep-iteration}} should show superset of fields
[KAFKA-6687] - Allow to read a topic multiple times
[KAFKA-6943] - Have option to shutdown KS cleanly if any threads crashes, or if all threads crash
[KAFKA-9023] - Producer NETWORK_EXCEPTION response should log more information
[KAFKA-9126] - Extend `StreamJoined` to allow more store configs
[KAFKA-9274] - Gracefully handle timeout exceptions on Kafka Streams
[KAFKA-9524] - Default window retention does not consider grace period
[KAFKA-9649] - Remove/Warn on use of TimeWindowedSerde with no specified window size
[KAFKA-10036] - Improve error message if user violates `Supplier` pattern
[KAFKA-10090] - Misleading warnings: The configuration was supplied but isn't a known config
[KAFKA-10092] - Remove unnecessary enum modifier in NioEchoServer
[KAFKA-10438] - Lazy initialization of record header to reduce memory usage in validating records
[KAFKA-10473] - Website is missing docs on JMX metrics for partition size-on-disk (kafka.log:type=Log,name=*)
[KAFKA-10497] - Convert group coordinator metadata schemas to use generated protocol
[KAFKA-10525] - Emit JSONs with new auto-generated schema
[KAFKA-10607] - Ensure the error counts contains the NONE
[KAFKA-10629] - TopologyTestDriver should not require a Properties arg
[KAFKA-10648] - Add Prefix Scan support to State Stores
[KAFKA-10673] - ConnectionQuotas should cache interbroker listener name
[KAFKA-10702] - Slow replication of empty transactions
[KAFKA-10720] - Add note to Transformation docs to avoid mutating records or their members
[KAFKA-10729] - KIP-482: Bump remaining RPC's to use tagged fields
[KAFKA-10736] - Convert transaction coordinator metadata schemas to use generated protocol
[KAFKA-10740] - Replace OffsetsForLeaderEpochRequest.PartitionData with automated protocol
[KAFKA-10768] - Make ByteBufferInputStream.read(byte[], int, int) to follow the contract
[KAFKA-10770] - Move Metrics#getTags to MetricsUtils so it can be reused by ConnectMetrics
[KAFKA-10794] - Replica leader election is too slow in the case of too many partitions
[KAFKA-10802] - Spurious log message when starting consumers
[KAFKA-10810] - Add a replace thread option to the streams uncaught exception handler
[KAFKA-10818] - Skip conversion to `Struct` when serializing generated requests/responses
[KAFKA-10833] - KIP-661: Expose task configurations in Connect REST API
[KAFKA-10834] - Remove redundant type casts in Connect
[KAFKA-10835] - Replace Runnable and Callable overrides with lambdas in Connect
[KAFKA-10839] - Improve consumer group coordinator unavailable message
[KAFKA-10841] - LogReadResult should be able to converted to FetchPartitionData
[KAFKA-10850] - Use primitive type to replace deprecated 'new Integer' from BrokerToControllerRequestThreadTest
[KAFKA-10851] - Add Cluster Describe API
[KAFKA-10852] - AlterIsr should not be throttled
[KAFKA-10874] - Fix flaky ClientQuotasRequestTest.testAlterIpQuotasRequest
[KAFKA-10878] - testReadWhenOptionalDataMissingAtTheEndIsNotTolerated/testReadWithMissingNonOptionalExtraDataAtTheEnd should check the error message
[KAFKA-12156] - Document consequences of single threaded response handling
[KAFKA-12180] - Implement the KIP-631 message generator changes
[KAFKA-12183] - Add the KIP-631 metadata record definitions
[KAFKA-12189] - ShellTest can replace 'assumeTrue' by (junit 5) conditional test
[KAFKA-12191] - SslTransportTls12Tls13Test can replace 'assumeTrue' by (junit 5) conditional test
[KAFKA-12206] - o.a.k.common.Uuid should implement Comparable
[KAFKA-12208] - Rename AdminManager to ZkAdminManager
[KAFKA-12246] - Remove redundant suppression in KafkaAdminClient
[KAFKA-12247] - Make removeStreamThread work better with static membership
[KAFKA-12310] - Update zookeeper to 3.5.9
[KAFKA-12327] - Remove MethodHandle usage in CompressionType
[KAFKA-12365] - Block APIs not supported by kip-500 broker/controller (for now)
[KAFKA-12394] - Consider topic id existence and authorization errors
[KAFKA-12400] - Upgrade jetty to fix CVE-2020-27223

Bug

[KAFKA-4748] - Need a way to shutdown all workers in a Streams application at the same time
[KAFKA-7987] - a broker's ZK session may die on transient auth failure
[KAFKA-8460] - Flaky Test PlaintextConsumerTest#testLowMaxFetchSizeForRequestAndPartition
[KAFKA-9263] - The new hw is added to incorrect log when ReplicaAlterLogDirsThread is replacing log (fix PlaintextAdminIntegrationTest.testAlterReplicaLogDirs)
[KAFKA-9393] - DeleteRecords may cause extreme lock contention for large partition directories
[KAFKA-9831] - Failing test: EosIntegrationTest.shouldNotViolateEosIfOneTaskFailsWithState[exactly_once_beta]
[KAFKA-9892] - Producer state snapshot needs to be forced to disk
[KAFKA-10017] - Flaky Test EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta
[KAFKA-10021] - When reading to the end of the config log, check if fetch.max.wait.ms is greater than worker.sync.timeout.ms
[KAFKA-10101] - recovery point is advanced without flushing the data after recovery
[KAFKA-10160] - Kafka MM2 consumer configuration
[KAFKA-10192] - Flaky test BlockingConnectorTest#testBlockInConnectorStop
[KAFKA-10366] - TimeWindowedDeserializer doesn't allow users to set a custom window size
[KAFKA-10413] - rebalancing leads to unevenly balanced connectors
[KAFKA-10417] - suppress() with cogroup() throws ClassCastException
[KAFKA-10454] - Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
[KAFKA-10460] - ReplicaListValidator format checking is incomplete
[KAFKA-10469] - Respect logging hierarchy (KIP-676)
[KAFKA-10470] - zstd decompression with small batches is slow and causes excessive GC
[KAFKA-10471] - TimeIndex handling may cause data loss in certain back to back failure
[KAFKA-10565] - Console producer displays interactive prompt even when input is not interactive
[KAFKA-10604] - The StreamsConfig.STATE_DIR_CONFIG's default value does not reflect the JVM parameter or OS-specific settings
[KAFKA-10656] - NetworkClient.java: print out the feature flags received at DEBUG level, as well as the other version information
[KAFKA-10658] - ErrantRecordReporter.report always return completed future even though the record is not sent to DLQ topic yet
[KAFKA-10671] - partition.assignment.strategy documentation does not include all options
[KAFKA-10678] - Re-deploying Streams app causes rebalance and task migration
[KAFKA-10685] - --to-datetime passed to kafka-consumer-groups interpreting microseconds wrong
[KAFKA-10689] - Assignor can't determine number of partitions on FJK with upstream windowed repartition
[KAFKA-10693] - Tests instantiate QuotaManagers without closing the managers in teardown
[KAFKA-10705] - Avoid World Readable RocksDB
[KAFKA-10710] - MirrorMaker 2 creates all combinations of herders
[KAFKA-10716] - Streams processId is unstable across restarts resulting in task mass migration
[KAFKA-10722] - Timestamped store is used even if not desired
[KAFKA-10723] - LogManager leaks internal thread pool activity during shutdown
[KAFKA-10727] - Kafka clients throw AuthenticationException during Kerberos re-login
[KAFKA-10763] - Task already exists error on same worker due to skip removal of tasks
[KAFKA-10776] - JMX metric RequestsPerSec requires API version to access
[KAFKA-10778] - Stronger log fencing after write failure
[KAFKA-10779] - Reassignment tool sets throttles incorrectly when overriding a reassignment
[KAFKA-10792] - Source tasks can block herder thread by hanging during stop
[KAFKA-10793] - Race condition in FindCoordinatorFuture permanently severs connection to group coordinator
[KAFKA-10798] - Failed authentication delay doesn't work with some SASL authentication failures
[KAFKA-10803] - Skip improper dynamic configs while initialization and include the rest correct ones
[KAFKA-10811] - System exit from MirrorConnectorsIntegrationTest#testReplication
[KAFKA-10832] - Recovery logic is using incorrect ProducerStateManager instance when updating producers
[KAFKA-10846] - FileStreamSourceTask buffer can grow without bound
[KAFKA-10895] - Basic auth extension's JAAS config can be corrupted by other plugins
[KAFKA-12152] - Idempotent Producer does not reset the sequence number of partitions without in-flight batches
[KAFKA-12153] - follower can hit OffsetOutOfRangeException during truncation
[KAFKA-12160] - KafkaStreams configs are documented incorrectly
[KAFKA-12165] - org.apache.kafka.common.quota classes omitted from Javadoc
[KAFKA-12185] - Flaky Test StoreQueryIntegrationTest#shouldQueryAllStalePartitionStores
[KAFKA-12190] - Failure on Windows due to an UnsupportedOperationException when StateDirectory sets file permissions
[KAFKA-12193] - Re-resolve IPs when a client is disconnected
[KAFKA-12195] - Fix synchronization issue happening in KafkaStreams (related to flaky AdjustStreamThreadCountTest)
[KAFKA-12211] - NoSuchFileException will be thrown if hasPersistentStores is false when creating stateDir
[KAFKA-12219] - Potential race condition in InMemoryKeyValueStore
[KAFKA-12233] - Align the length passed to FileChannel by `FileRecords.writeTo`
[KAFKA-12235] - ZkAdminManager.describeConfigs returns no config when 2+ configuration keys are specified
[KAFKA-12254] - MirrorMaker 2.0 creates destination topic with default configs
[KAFKA-12259] - Consolidated Status endpoint returns 500 when config provider can't find a config
[KAFKA-12268] - System tests broken because consumer returns early without records
[KAFKA-12270] - Kafka Connect may fail a task when racing to create topic
[KAFKA-12272] - Kafka Streams metric commit-latency-max and commit-latency-avg is always 0
[KAFKA-12321] - the comparison function for uuid type should be 'equals' rather than '=='
[KAFKA-12323] - Record timestamps not populated in event
[KAFKA-12324] - Upgrade jetty to fix CVE-2020-27218
[KAFKA-12326] - MM2 fails to start to due missing required configuration "bootstrap.servers"
[KAFKA-12339] - Add retry to admin client's listOffsets
[KAFKA-12340] - Recent change to use SharedTopicAdmin results in potential resource leak in deprecated backing store constructors
[KAFKA-12341] - Duplicate javassist library found in broker classpath
[KAFKA-12343] - Recent change to use SharedTopicAdmin in KakfkaBasedLog fails with AK 0.10.x brokers
[KAFKA-12357] - Do not inline methods from the scala package by default
[KAFKA-12367] - RaftReplicaManager needs to propagate isr version to Partition
[KAFKA-12374] - Add missing config sasl.mechanism.controller.protocol
[KAFKA-12375] - ReplaceStreamThread creates a new consumer with the same name as the one it's replacing
[KAFKA-12376] - Use scheduleAtomicAppend for records that need to be atomic
[KAFKA-12381] - Incompatible change in verifiable_producer.log in 2.8
[KAFKA-12382] - Create KIP-500 README for the 2.8 release
[KAFKA-12389] - Upgrade of netty-codec due to CVE-2021-21290
[KAFKA-12393] - Document multi-tenancy considerations
[KAFKA-12435] - Several streams-test-utils classes missing from javadoc
[KAFKA-12462] - Threads in PENDING_SHUTDOWN entering a rebalance can cause an illegal state exception
[KAFKA-12474] - Worker can die if unable to write new session key
[KAFKA-12500] - Cache memory is leaked after removing/replacing a StreamThread
[KAFKA-12503] - Resizing the thread cache in a non thread safe way can cause records to be redirected throughout the topology
[KAFKA-12508] - Emit-on-change tables may lose updates on error or restart in at_least_once
[KAFKA-12523] - Need to improve handling of TimeoutException when committing offsets
[KAFKA-12537] - Single Threaded applications will not work with SHUTDOWN_APPLICATION
[KAFKA-12557] - org.apache.kafka.clients.admin.KafkaAdminClientTest#testClientSideTimeoutAfterFailureToReceiveResponse intermittently hangs indefinitely
[KAFKA-12583] - Upgrade of netty-codec due to CVE-2021-21295
[KAFKA-12593] - Some Scala, Python, and Gradle files contain the wrong license header
[KAFKA-12602] - The LICENSE and NOTICE files don't list everything they should
[KAFKA-12655] - CVE-2021-28165 - Upgrade jetty to 9.4.39
[KAFKA-12658] - bin/kafka-metadata-shell.sh cannot find or load main class org.apache.kafka.shell.MetadataShell

Task

[KAFKA-8744] - Add Support to Scala API for KIP-307 and KIP-479
[KAFKA-10554] - Perform follower truncation based on epoch offsets returned in Fetch response
[KAFKA-10583] - Thread-safety of AdminClient is not documented
[KAFKA-10628] - Follow-up: Remove all unnecessary dummy TopologyTestDriver configs

Test

[KAFKA-10304] - Revisit and improve the tests of MirrorMaker 2
[KAFKA-10754] - Fix flaky shouldShutdownSingleThreadApplication test
[KAFKA-10854] - Fix flaky kafka.network.SocketServerTest.testConnectionRatePerIp test
[KAFKA-12194] - fix flaky AdjustStreamThreadCountTest tests
[KAFKA-12229] - Fix flaky MirrorSourceConnectorTest.testRefreshTopicPartitionsTopicOnTargetFirst test
[KAFKA-12273] - InterBrokerSendThread#pollOnce throws FatalExitError even though it is shutdown correctly

Sub-task

[KAFKA-7340] - Migrate clients module to JUnit 5
[KAFKA-7341] - Migrate core module to JUnit 5
[KAFKA-9628] - Replace Produce request/response with automated protocol
[KAFKA-9630] - Replace OffsetsForLeaderEpoch request/response with automated protocol
[KAFKA-10024] - Add dynamic configuration and enforce quota for per-IP connection rate limits (KIP-612, part 2)
[KAFKA-10181] - Create Envelope RPC and redirection template for configuration change RPCs
[KAFKA-10624] - [Easy] FeatureZNodeStatus should use sealed trait instead of Enumeration
[KAFKA-10692] - Rename broker master key config for KIP 681
[KAFKA-10747] - Implement ClientQuota APIs for altering and describing IP entity quotas
[KAFKA-10748] - Add IP connection rate throttling metric
[KAFKA-10815] - EosTestDriver#verifyAllTransactionFinished should break loop if all partitions are verified
[KAFKA-10866] - Add fetched metadata to ConsumerRecords
[KAFKA-10867] - Implement improved semantics using the ConsumerRecords meta
[KAFKA-10887] - Migrate log4j-appender module to JUnit 5
[KAFKA-10893] - Increase target_messages_per_sec of ReplicaScaleTest to reduce the run time
[KAFKA-12171] - Migrate streams:test-utils module to JUnit 5
[KAFKA-12172] - Migrate streams:examples module to JUnit 5
[KAFKA-12175] - Migrate generator module to JUnit 5
[KAFKA-12196] - Migrate connect:api module to JUnit 5
[KAFKA-12197] - Migrate connect:transforms module to JUnit 5
[KAFKA-12198] - Migrate connect:json module to JUnit 5
[KAFKA-12200] - Migrate connect:file module to JUnit 5
[KAFKA-12201] - Migrate connect:basic-auth-extension module to JUnit 5
[KAFKA-12202] - Migrate connect:mirror module to JUnit 5
[KAFKA-12203] - Migrate connect:mirror-client module to JUnit 5
[KAFKA-12204] - Implement DescribeCluster API on the broker
[KAFKA-12210] - AdminClient should use DescribeCluster API when available
[KAFKA-12212] - Bump Metadata API version to remove `ClusterAuthorizedOperations` fields
[KAFKA-12238] - Implement DescribeProducers API