Release Notes - Kafka - Version 2.2.0

Below is a summary of the JIRA issues addressed in the 2.2.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at http://kafka.apache.org/documentation.html.

Improvement

• [KAFKA-4217] - KStream.transform equivalent of flatMap
• [KAFKA-4453] - add request prioritization
• [KAFKA-4850] - RocksDb cannot use Bloom Filters
• [KAFKA-5692] - Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
• [KAFKA-5994] - Improve transparency of broker user ACL misconfigurations
• [KAFKA-6431] - Lock contention in Purgatory
• [KAFKA-6567] - KStreamWindowReduce can be replaced by KStreamWindowAggregate
• [KAFKA-6774] - Improve default groupId behavior in consumer
• [KAFKA-6928] - StreamsPartitionAssignor is double retrying within InternalTopicManager
• [KAFKA-7024] - Rocksdb state directory should be created before opening the DB
• [KAFKA-7054] - Kafka describe command should throw topic doesn't exist exception.
• [KAFKA-7096] - Consumer should drop the data for unassigned topic partitions
• [KAFKA-7131] - Update release script to generate announcement email text
• [KAFKA-7235] - Use brokerZkNodeVersion to prevent broker from processing outdated controller request
• [KAFKA-7266] - Fix MetricsTest test flakiness
• [KAFKA-7352] - KIP-368: Allow SASL Connections to Periodically Re-Authenticate
• [KAFKA-7367] - Streams should not create state store directories unless they are needed
• [KAFKA-7389] - Upgrade spotBugs for Java 11 support
• [KAFKA-7391] - Introduce close(Duration) to Producer and AdminClient instead of close(long, TimeUnit)
• [KAFKA-7402] - Kafka Streams should implement AutoCloseable where appropriate
• [KAFKA-7413] - Replace slave terminology with follower in website
• [KAFKA-7418] - Add '--help' option to all available Kafka CLI commands (KIP-374)
• [KAFKA-7420] - Global stores should be guarded as read-only for regular tasks
• [KAFKA-7433] - Introduce broker options in TopicCommand to use AdminClient
• [KAFKA-7446] - Better error message to explain the upper limit of TimeWindow
• [KAFKA-7461] - Connect Values converter should have coverage of logical types
• [KAFKA-7478] - Reduce OAuthBearerLoginModule verbosity
• [KAFKA-7510] - KStreams RecordCollectorImpl leaks data to logs on error
• [KAFKA-7524] - Recommend Scala 2.12 and use it for development
• [KAFKA-7528] - Standardize on Min/Avg/Max Kafka metrics' default value
• [KAFKA-7537] - Only include live brokers in the UpdateMetadataRequest sent to existing brokers if there is no change in the partition states
• [KAFKA-7567] - Clean up internal metadata usage for consistency and extensibility
• [KAFKA-7568] - Return leader epoch in ListOffsets responses
• [KAFKA-7584] - StreamsConfig throws ClassCastException if max.in.flight.request.per.connect is specified as String
• [KAFKA-7610] - Detect consumer failures in initial JoinGroup
• [KAFKA-7641] - Add `consumer.group.max.size` to cap consumer metadata size on broker
• [KAFKA-7671] - A KStream/GlobalKTable join shouldn't reset the repartition flag
• [KAFKA-7673] - Upgrade RocksDB to include fix for WinEnvIO::GetSectorSize
• [KAFKA-7687] - Print batch level information in DumpLogSegments when deep iterating
• [KAFKA-7719] - Improve fairness in SocketServer processors
• [KAFKA-7734] - Metrics tags should use LinkedHashMap to guarantee ordering
• [KAFKA-7738] - Track partition leader epochs in client metadata
• [KAFKA-7758] - When Naming a Repartition Topic with Aggregations Reuse Repartition Graph Node for Multiple Operations
• [KAFKA-7762] - KafkaConsumer uses old API in the javadocs
• [KAFKA-7798] - Expose embedded client context from KafkaStreams threadMetadata
• [KAFKA-7804] - Update the docs for KIP-377
• [KAFKA-7829] - Javadoc should show that AdminClient.alterReplicaLogDirs() is supported in Kafka 1.1.0 or later
• [KAFKA-7834] - Extend collected logs in system test services to include heap dumps
• [KAFKA-7837] - maybeShrinkIsr may not reflect OfflinePartitions immediately
• [KAFKA-7838] - improve logging in Partition.maybeShrinkIsr()
• [KAFKA-7880] - KafkaConnect should standardize worker thread name
• [KAFKA-7916] - Streams store cleanup: unify wrapping
• [KAFKA-8040] - Streams needs to handle timeout in initTransactions

Bug

• [KAFKA-2334] - Prevent HW from going back during leader failover
• [KAFKA-3832] - Kafka Connect's JSON Converter never outputs a null value
• [KAFKA-4468] - Correctly calculate the window end timestamp after read from state stores
• [KAFKA-5117] - Kafka Connect REST endpoints reveal Password typed values
• [KAFKA-5462] - Add a configuration for users to specify a template for building a custom principal name
• [KAFKA-5950] - AdminClient should retry based on returned error codes
• [KAFKA-6478] - kafka-run-class.bat fails if CLASSPATH contains spaces
• [KAFKA-6833] - KafkaProducer throws "Invalid partition given with record" exception
• [KAFKA-6970] - Kafka streams lets the user call init() and close() on a state store, when inside Processors
• [KAFKA-7037] - delete topic command replaces '+' from the topic name which leads incorrect topic deletion
• [KAFKA-7051] - Improve the efficiency of the ReplicaManager when there are many partitions
• [KAFKA-7110] - Windowed changelog keys not deserialized properly by TimeWindowedSerde
• [KAFKA-7165] - Error while creating ephemeral at /brokers/ids/BROKER_ID
• [KAFKA-7181] - Kafka Streams State stuck in rebalancing after one of the StreamThread encounters IllegalStateException
• [KAFKA-7253] - The connector type responded by worker is always null when creating connector
• [KAFKA-7288] - Transient failure in SslSelectorTest.testCloseConnectionInClosingState
• [KAFKA-7312] - Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
• [KAFKA-7412] - Bug prone response from producer.send(ProducerRecord, Callback) if Kafka broker is not running
• [KAFKA-7443] - OffsetOutOfRangeException in restoring state store from changelog topic when start offset of local checkpoint is smaller than that of changelog topic
• [KAFKA-7449] - Kafka console consumer is not sending topic to deserializer
• [KAFKA-7476] - SchemaProjector is not properly handling Date-based logical types
• [KAFKA-7518] - FutureRecordMetadata.get deadline calculation from timeout is not using timeunit
• [KAFKA-7532] - Missing stringification in error message "Removed ArrayBuffer() from list of shutting down brokers. (kafka.controller.KafkaController)"
• [KAFKA-7536] - TopologyTestDriver cannot pre-populate KTable or GlobalKTable
• [KAFKA-7549] - Old ProduceRequest with zstd compression does not return error to client
• [KAFKA-7557] - optimize LogManager.truncateFullyAndStartAt()
• [KAFKA-7560] - PushHttpMetricsReporter should not convert metric value to double
• [KAFKA-7561] - Console Consumer - system test fails
• [KAFKA-7605] - Flaky Test `SaslMultiMechanismConsumerTest.testCoordinatorFailover`
• [KAFKA-7616] - MockConsumer can return ConsumerRecords objects with a non-empty map but no records
• [KAFKA-7620] - ConfigProvider is broken for KafkaConnect when TTL is not null
• [KAFKA-7655] - Metadata spamming requests from Kafka Streams under some circumstances, potential DOS
• [KAFKA-7657] - Invalid reporting of stream state in Kafka streams application
• [KAFKA-7660] - Stream Metrics - Memory Analysis
• [KAFKA-7672] - The local state not fully restored after KafkaStream rebalanced, resulting in data loss
• [KAFKA-7678] - Failed to close producer due to java.lang.NullPointerException
• [KAFKA-7692] - updateFirstUnstableOffset NPE due to sequenceId overflow in ProducerStateManager.append
• [KAFKA-7693] - "IllegalArgumentException: Invalid negative sequence number used" in Kafka Client
• [KAFKA-7697] - Possible deadlock in kafka.cluster.Partition
• [KAFKA-7702] - Prefixed ACLs don't work with single character prefix
• [KAFKA-7704] - kafka.server.ReplicaFetechManager.MaxLag.Replica metric is reported incorrectly
• [KAFKA-7709] - ConcurrentModificationException occurs when iterating through multiple partitions in Sender.getExpiredInflightBatches
• [KAFKA-7712] - Handle exceptions from immediately connected channels in Selector
• [KAFKA-7741] - Bad dependency via SBT
• [KAFKA-7742] - DelegationTokenCache#hmacIdCache entry is not cleared when a token is removed using removeToken(String tokenId) API.
• [KAFKA-7752] - zookeeper-security-migration.sh does not remove ACL on kafka-acl-extended
• [KAFKA-7755] - Kubernetes - Kafka clients are resolving DNS entries only one time
• [KAFKA-7759] - Disable WADL output in Connect REST API
• [KAFKA-7768] - Java doc link 404
• [KAFKA-7781] - Add validation check for Topic retention.ms property
• [KAFKA-7786] - Fast update of leader epoch may stall partition fetching due to FENCED_LEADER_EPOCH
• [KAFKA-7789] - SSL-related unit tests hang when run on Fedora 29
• [KAFKA-7808] - AdminClient#describeTopics should not throw InvalidTopicException if topic name is not found
• [KAFKA-7840] - Documentation for cleanup.policy is out of date
• [KAFKA-7873] - KafkaBasedLog's consumer should always seek to beginning when starting
• [KAFKA-7884] - Docs for message.format.version and log.message.format.version show invalid (corrupt?) "valid values"
• [KAFKA-7890] - Invalidate ClusterConnectionState cache for a broker if the hostname of the broker changes.
• [KAFKA-7895] - Ktable supress operator emitting more than one record for the same key per window
• [KAFKA-7897] - Invalid use of epoch cache with old message format versions
• [KAFKA-7902] - SASL/OAUTHBEARER can become unable to connect: javax.security.sasl.SaslException: Unable to find OAuth Bearer token in Subject's private credentials (size=2)
• [KAFKA-7909] - Ensure timely rebalance completion after pending members rejoin or fail
• [KAFKA-7915] - SASL authentication failures may return sensitive data to client
• [KAFKA-7920] - Do not permit zstd use until inter.broker.protocol.version is updated to 2.1
• [KAFKA-7935] - UNSUPPORTED_COMPRESSION_TYPE if ReplicaManager.getLogConfig returns None
• [KAFKA-7938] - Flaky Test DeleteConsumerGroupsTest#testDeleteCmdWithShortInitialization
• [KAFKA-7940] - Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
• [KAFKA-7945] - ExpiringCredentialRefreshingLogin - timeout value is negative
• [KAFKA-7977] - Flaky Test ReassignPartitionsClusterTest#shouldOnlyThrottleMovingReplicas
• [KAFKA-7979] - Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords
• [KAFKA-7980] - Flaky Test SocketServerTest#testConnectionRateLimit
• [KAFKA-7999] - Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
• [KAFKA-8002] - Replica reassignment to new log dir may not complete if future and current replicas segment files have different base offsets
• [KAFKA-8011] - Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
• [KAFKA-8012] - NullPointerException while truncating at high watermark can crash replica fetcher thread
• [KAFKA-8018] - Flaky Test SaslSslAdminClientIntegrationTest#testLegacyAclOpsNeverAffectOrReturnPrefixed
• [KAFKA-8022] - Flaky Test RequestQuotaTest#testExemptRequestTime
• [KAFKA-8038] - Flaky Test SslTransportLayerTest#testCloseSsl
• [KAFKA-8039] - Flaky Test SaslAuthenticatorTest#testCannotReauthenticateAgainFasterThanOneSecond
• [KAFKA-8061] - Handle concurrent ProducerId reset and call to Sender thread shutdown
• [KAFKA-8065] - Forwarding modified timestamps does not reset timestamp correctly
• [KAFKA-8069] - Committed offsets get cleaned up right after the coordinator loading them back from __consumer_offsets in broker with old inter-broker protocol version (< 2.2)
• [KAFKA-8070] - System test ConsumerGroupCommandTest fails intermittently with SSL

Task

• [KAFKA-7259] - Remove deprecated ZKUtils usage from ZkSecurityMigrator
• [KAFKA-7431] - Clean up connect unit tests code
• [KAFKA-7503] - Integration Test Framework for Connect
• [KAFKA-7551] - Refactor to create both producer & consumer in Worker
• [KAFKA-7617] - Document security primitives

Test

• [KAFKA-7799] - Fix flaky test RestServerTest.testCORSEnabled

Sub-task

• [KAFKA-1120] - Controller could miss a broker state change
• [KAFKA-4544] - Add system tests for delegation token based authentication
• [KAFKA-5272] - Improve validation for Alter Configs (KIP-133)
• [KAFKA-5503] - Idempotent producer ignores shutdown while fetching ProducerId
• [KAFKA-6036] - Enable logical materialization to physical materialization
• [KAFKA-7439] - Replace EasyMock and PowerMock with Mockito in clients module
• [KAFKA-7612] - Fix javac warnings and enable warnings as errors
• [KAFKA-7651] - Flaky test SaslSslAdminClientIntegrationTest.testMinimumRequestTimeouts
• [KAFKA-7824] - Require member.id for initial join group request