Release Notes - Kafka - Version 2.5.1

Below is a summary of the JIRA issues addressed in the 2.5.1 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

Improvement

• [KAFKA-8938] - Connect - Improve Memory Allocations During Struct Validation
• [KAFKA-9127] - Needless group coordination overhead for GlobalKTables
• [KAFKA-9216] - Enforce connect internal topic configuration at startup
• [KAFKA-9419] - Integer Overflow Possible with CircularIterator
• [KAFKA-9468] - config.storage.topic partition count issue is hard to debug
• [KAFKA-9919] - Add logging to KafkaBasedLog
• [KAFKA-9987] - Improve sticky partition assignor algorithm

Bug

• [KAFKA-6647] - KafkaStreams.cleanUp creates .lock file in directory its trying to clean (Windows OS)
• [KAFKA-8391] - Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
• [KAFKA-8661] - Flaky Test RebalanceSourceConnectorsIntegrationTest#testStartTwoConnectors
• [KAFKA-8672] - RebalanceSourceConnectorsIntegrationTest#testReconfigConnector
• [KAFKA-8869] - Map taskConfigs in KafkaConfigBackingStore grows monotonically despite of task removals
• [KAFKA-9298] - Reuse of a mapped stream causes an Invalid Topology
• [KAFKA-9472] - Reducing number of tasks for connector causes deleted tasks to show as UNASSIGNED
• [KAFKA-9533] - JavaDocs of KStream#ValueTransform incorrect
• [KAFKA-9537] - Abstract transformations in configurations cause unfriendly error message.
• [KAFKA-9540] - Application getting "Could not find the standby task 0_4 while closing it" error
• [KAFKA-9543] - Consumer offset reset after new segment rolling
• [KAFKA-9570] - SSL cannot be configured for Connect in standalone mode
• [KAFKA-9583] - OffsetsForLeaderEpoch requests are sometimes not sent to the leader of partition
• [KAFKA-9625] - Unable to Describe broker configurations that have been set via IncrementalAlterConfigs
• [KAFKA-9633] - ConfigProvider.close() not called
• [KAFKA-9654] - ReplicaAlterLogDirsThread can't be created again if the previous ReplicaAlterLogDirsThreadmeet encounters leader epoch error
• [KAFKA-9667] - Connect JSON serde strip trailing zeros
• [KAFKA-9669] - Kafka 2.4.0 Chokes on Filebeat 5.6 Produced Data
• [KAFKA-9675] - RocksDB metrics reported always at zero
• [KAFKA-9704] - z/OS won't let us resize file when mmap
• [KAFKA-9718] - Don't log passwords for AlterConfigs requests in request logs
• [KAFKA-9724] - Consumer wrongly ignores fetched records "since it no longer has valid position"
• [KAFKA-9739] - StreamsBuilder.build fails with StreamsException "Found a null keyChangingChild node for OptimizableRepartitionNode"
• [KAFKA-9763] - Recent changes to Connect's InsertField will fail to inject field on key of tombstone record
• [KAFKA-9768] - rest.advertised.listener configuration is not handled properly by the worker
• [KAFKA-9802] - Fix flaky system test `TransactionsTest.test_transactions`
• [KAFKA-9826] - Log cleaning repeatedly picks same segment with no effect when first dirty offset is past start of active segment
• [KAFKA-9830] - DeadLetterQueueReporter leaks KafkaProducer instance
• [KAFKA-9835] - Race condition with concurrent write allows reads above high watermark
• [KAFKA-9839] - IllegalStateException on metadata update when broker learns about its new epoch after the controller
• [KAFKA-9841] - Connector and Task duplicated when a worker join with old generation assignment
• [KAFKA-9845] - plugin.path property does not work with config provider
• [KAFKA-9848] - Avoid triggering scheduled rebalance delay when task assignment fails but Connect workers remain in the group
• [KAFKA-9849] - Fix issue with worker.unsync.backoff.ms creating zombie workers when incremental cooperative rebalancing is used
• [KAFKA-9851] - Revoking Connect tasks due to connectivity issues should also clear running assignment
• [KAFKA-9854] - Re-authenticating causes mismatched parse of response
• [KAFKA-9863] - update the deprecated --zookeeper option in the documentation into --bootstrap-server
• [KAFKA-9883] - Connect request to restart task can result in IllegalArgumentError: "uriTemplate" parameter is null
• [KAFKA-9888] - REST extensions can mutate connector configs in worker config state snapshot
• [KAFKA-9891] - Invalid state store content after task migration with exactly_once and standby replicas
• [KAFKA-9921] - Caching is not working properly with WindowStateStore when retaining duplicates
• [KAFKA-9922] - Update examples README
• [KAFKA-9925] - Non-key KTable Joining may result in duplicate schema name in confluence schema registry
• [KAFKA-9950] - MirrorMaker2 sharing of ConfigDef can lead to ConcurrentModificationException
• [KAFKA-9955] - Exceptions thrown from SinkTask::close shadow other exceptions
• [KAFKA-9969] - ConnectorClientConfigRequest is loaded in isolation and throws LinkageError
• [KAFKA-9985] - Sink connector consuming DLQ topic may exhaust broker
• [KAFKA-9992] - EmbeddedKafkaCluster.deleteTopicAndWait not working with kafka_2.13
• [KAFKA-9996] - upgrade zookeeper to 3.5.8 to address security vulnerabilities
• [KAFKA-10004] - ConfigCommand fails to find default broker configs without ZK
• [KAFKA-10029] - Selector.completedReceives should not be modified when channel is closed
• [KAFKA-10030] - Throw exception while fetching a key from a single partition
• [KAFKA-10049] - KTable-KTable Foreign Key join throwing Serialization Exception
• [KAFKA-10056] - Consumer metadata may use outdated groupSubscription that doesn't contain newly subscribed topics
• [KAFKA-10066] - TopologyTestDriver isn't taking record headers into account during de/serialization
• [KAFKA-10123] - Regression resetting offsets in consumer when fetching from old broker
• [KAFKA-10134] - High CPU issue during rebalance in Kafka consumer after upgrading to 2.5
• [KAFKA-10173] - BufferUnderflowException during Kafka Streams Upgrade
• [KAFKA-10212] - Describing a topic with the TopicCommand fails if unauthorised to use ListPartitionReassignments API
• [KAFKA-10221] - Backport fix for KAFKA-9603 to 2.5
• [KAFKA-10239] - The groupInstanceId field in DescribeGroup response should be ignorable
• [KAFKA-10254] - 100% cpu usage by kafkaConsumer poll , when broker can't be connect

Task

• [KAFKA-10185] - Streams should log summarized restoration information at info level

Test

• [KAFKA-10286] - Connect system tests should wait for workers to join group
• [KAFKA-10295] - ConnectDistributedTest.test_bounce should wait for graceful stop