Release Notes - Kafka - Version 2.1.0
Below is a summary of the JIRA issues addressed in the 2.1.0 release of Kafka. For full documentation of the
release, a guide to get started, and information about the project, see the Kafka
project site.
Note about upgrades: Please carefully review the
upgrade documentation for this release thoroughly
before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking
changes, performance changes, and any other changes that might impact your production deployment of Kafka.
The documentation for the most recent release can be found at
http://kafka.apache.org/documentation.html.
New Feature
- [KAFKA-7027] - Overloaded StreamsBuilder Build Method to Accept java.util.Properties
Improvement
- [KAFKA-4514] - Add Codec for ZStandard Compression
- [KAFKA-4932] - Add UUID Serde
- [KAFKA-5886] - Introduce delivery.timeout.ms producer config (KIP-91)
- [KAFKA-5928] - Avoid redundant requests to zookeeper when reassign topic partition
- [KAFKA-6195] - DNS alias support for secured connections
- [KAFKA-6432] - Lookup indices may cause unnecessary page fault
- [KAFKA-6437] - Streams does not warn about missing input topics, but hangs
- [KAFKA-6591] - Move check for super user in SimpleAclProvider before ACL evaluation
- [KAFKA-6684] - Support casting values with bytes schema to string
- [KAFKA-6751] - Make max.connections.per.ip.overrides a dynamic config
- [KAFKA-6753] - Speed up event processing on the controller
- [KAFKA-6761] - Reduce Kafka Streams Footprint
- [KAFKA-6863] - Kafka clients should try to use multiple DNS resolved IP addresses if the first one fails
- [KAFKA-6880] - Zombie replicas must be fenced
- [KAFKA-6884] - ConsumerGroupCommand should use new AdminClient
- [KAFKA-6914] - Kafka Connect - Plugins class should have a constructor that can take in parent ClassLoader
- [KAFKA-6923] - Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer
- [KAFKA-6926] - Reduce NPath exceptions in Connect
- [KAFKA-6950] - Add mechanism to delay response to failed client authentication
- [KAFKA-6978] - Make Streams Window retention time strict
- [KAFKA-6986] - Export Admin Client metrics through Stream Threads
- [KAFKA-6998] - Remove caching wrapper stores if cache-size is configured to zero bytes
- [KAFKA-6999] - Document read-write lock usage of caching enabled stores
- [KAFKA-7009] - Mute logger for reflections.org at the warn level in system tests
- [KAFKA-7019] - Reduction the contention between metadata update and metadata read operation
- [KAFKA-7101] - Session Window store should set topic policy `compact,cleanup`
- [KAFKA-7117] - Allow AclCommand to use AdminClient API
- [KAFKA-7126] - Reduce number of rebalance for large consumer groups after a topic is created
- [KAFKA-7131] - Update release script to generate announcement email text
- [KAFKA-7147] - Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file
- [KAFKA-7152] - replica should be in-sync if its LEO equals leader's LEO
- [KAFKA-7161] - KTable Reduce should check for invalid conditions
- [KAFKA-7169] - Add support for Custom SASL extensions in OAuth authentication
- [KAFKA-7198] - Enhance KafkaStreams start method javadoc
- [KAFKA-7207] - Make rate & total metrics documentation consistent
- [KAFKA-7210] - Add system test for log compaction
- [KAFKA-7211] - MM should handle timeouts in commitSync
- [KAFKA-7215] - Improve LogCleaner behavior on error
- [KAFKA-7222] - KIP-328: Add Window Grace Period (and deprecate Window Retention)
- [KAFKA-7264] - Initial Kafka support for Java 11
- [KAFKA-7266] - Fix MetricsTest test flakiness
- [KAFKA-7269] - KStream.merge is not documented
- [KAFKA-7277] - Migrate Streams API to Duration instead of longMs times
- [KAFKA-7278] - replaceSegments() should not call asyncDeleteSegment() for segments which have been removed from segments list
- [KAFKA-7285] - Streams should be more fencing-sensitive during task suspension under EOS
- [KAFKA-7313] - StopReplicaRequest should attempt to remove future replica for the partition only if future replica exists
- [KAFKA-7326] - Let KStream.print() to flush on each printed line
- [KAFKA-7332] - Improve error message when trying to produce message without key for compacted topic
- [KAFKA-7394] - Allow OffsetsForLeaderEpoch requests with topic describe ACL (KIP-320)
- [KAFKA-7395] - Add fencing to replication protocol (KIP-320)
- [KAFKA-7396] - KIP-365: Materialized, Serialized, Joined, Consumed and Produced with implicit Serde
- [KAFKA-7399] - FunctionConversions in Streams-Scala should be private
- [KAFKA-7406] - Naming Join and Grouping Repartition Topics
- [KAFKA-7414] - Do not fail broker on out of range offsets in replica fetcher
- [KAFKA-7429] - Enable dynamic key/truststore update with same filename/password
- [KAFKA-7430] - Improve Transformer interface JavaDoc
- [KAFKA-7437] - Store leader epoch in offset commit metadata
- [KAFKA-7441] - Allow LogCleanerManager.resumeCleaning() to be used concurrently
- [KAFKA-7454] - Use lazy allocation for SslTransportLayer buffers
- [KAFKA-7456] - Serde Inheritance in Streams DSL
- [KAFKA-7475] - print the actual cluster bootstrap address on authentication failures
- [KAFKA-7477] - Improve Streams close timeout semantics
- [KAFKA-7559] - ConnectStandaloneFileTest system tests do not pass
Bug
- [KAFKA-1420] - Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in unit tests
- [KAFKA-3097] - Acls for PrincipalType User are case sensitive
- [KAFKA-3514] - Stream timestamp computation needs some further thoughts
- [KAFKA-4682] - Committed offsets should not be deleted if a consumer is still active (KIP-211)
- [KAFKA-4950] - ConcurrentModificationException when iterating over Kafka Metrics
- [KAFKA-4994] - Fix findbugs warning about OffsetStorageWriter#currentFlushId
- [KAFKA-5037] - Infinite loop if all input topics are unknown at startup
- [KAFKA-5066] - KafkaMetricsConfig properties and description notably missing from documentation
- [KAFKA-5098] - KafkaProducer.send() blocks and generates TimeoutException if topic name has illegal char
- [KAFKA-5638] - Inconsistency in consumer group related ACLs
- [KAFKA-5690] - kafka-acls command should be able to list per principal
- [KAFKA-5975] - No response when deleting topics and delete.topic.enable=false
- [KAFKA-6123] - Give client MetricsReporter auto-generated client.id
- [KAFKA-6327] - IllegalArgumentException in RocksDB when RocksDBException being generated
- [KAFKA-6343] - OOM as the result of creation of 5k topics
- [KAFKA-6415] - KafkaLog4jAppender deadlocks when logging from producer network thread
- [KAFKA-6587] - Kafka Streams hangs when not able to access internal topics
- [KAFKA-6620] - Documentation about "exactly_once" doesn't mention "transaction.state.log.min.isr"
- [KAFKA-6648] - Fetcher.getTopicMetadata() should return all partitions for each requested topic
- [KAFKA-6859] - Follower should not send OffsetForLeaderEpoch for undefined leader epochs
- [KAFKA-6966] - Extend `TopologyDescription.Sink` to return `TopicNameExtractor`
- [KAFKA-6991] - Connect Rest Extension Plugin issue with Class Loader
- [KAFKA-7031] - Kafka Connect API module depends on Jersey
- [KAFKA-7043] - Connect isolation whitelist does not include new primitive converters (KIP-305)
- [KAFKA-7044] - kafka-consumer-groups.sh NullPointerException describing round robin or sticky assignors
- [KAFKA-7047] - Connect isolation whitelist does not include SimpleHeaderConverter
- [KAFKA-7056] - Connect's new numeric converters should be in a different package
- [KAFKA-7058] - ConnectSchema#equals() broken for array-typed default values
- [KAFKA-7068] - ConfigTransformer doesn't handle null values
- [KAFKA-7072] - Kafka Streams may drop rocksb window segments before they expire
- [KAFKA-7080] - WindowStoreBuilder incorrectly initializes CachingWindowStore
- [KAFKA-7097] - VerifiableProducer does not work properly with --message-create-time argument
- [KAFKA-7103] - Use bulkloading for RocksDBSegmentedBytesStore during init
- [KAFKA-7119] - Intermittent test failure with GSSAPI authentication failure
- [KAFKA-7128] - Lagging high watermark can lead to committed data loss after ISR expansion
- [KAFKA-7134] - KafkaLog4jAppender - Appender exceptions are propagated to caller
- [KAFKA-7141] - kafka-consumer-group doesn't describe existing group
- [KAFKA-7144] - Kafka Streams doesn't properly balance partition assignment
- [KAFKA-7158] - Duplicates when searching kafka stream state store with caching
- [KAFKA-7164] - Follower should truncate after every leader epoch change
- [KAFKA-7180] - In testHWCheckpointWithFailuresSingleLogSegment, wait until server1 has joined the ISR before shutting down server2
- [KAFKA-7192] - State-store can desynchronise with changelog
- [KAFKA-7196] - Remove heartbeat delayed operation for those removed consumers at the end of each rebalance
- [KAFKA-7216] - Exception while running kafka-acls.sh from 1.0 env on target Kafka env with 1.1.1
- [KAFKA-7225] - Kafka Connect ConfigProvider not invoked before validation
- [KAFKA-7240] - -total metrics in Streams are incorrect
- [KAFKA-7242] - Externalized secrets are revealed in task configuration
- [KAFKA-7247] - A link to Apache BookKeeper project is broken
- [KAFKA-7250] - Kafka-Streams-Scala DSL transform shares transformer instance
- [KAFKA-7255] - Timing issue in SimpleAclAuthorizer with concurrent create/update
- [KAFKA-7261] - Request and response total metrics record bytes instead of request count
- [KAFKA-7280] - ConcurrentModificationException in FetchSessionHandler in heartbeat thread
- [KAFKA-7284] - Producer getting fenced may cause Streams to shut down
- [KAFKA-7286] - Loading offsets and group metadata hangs with large group metadata records
- [KAFKA-7287] - Set open ACL permissions for old consumer znode path
- [KAFKA-7288] - Transient failure in SslSelectorTest.testCloseConnectionInClosingState
- [KAFKA-7301] - KTable to KTable join invocation does not resolve in Scala DSL
- [KAFKA-7311] - Sender should reset next batch expiry time between poll loops
- [KAFKA-7316] - Use of filter method in KTable.scala may result in StackOverflowError
- [KAFKA-7322] - Fix race condition between log cleaner thread and log retention thread when topic cleanup policy is updated
- [KAFKA-7324] - NPE due to lack of SASLExtensions in SASL/OAUTHBEARER
- [KAFKA-7333] - Protocol changes for KIP-320
- [KAFKA-7347] - Wrong error code returned for OffsetsForLeaderEpoch from non-replica
- [KAFKA-7353] - Connect logs 'this' for anonymous inner classes
- [KAFKA-7354] - Fix IdlePercent and NetworkProcessorAvgIdlePercent metric calculation
- [KAFKA-7366] - topic level segment.bytes and segment.ms not taking effect immediately
- [KAFKA-7369] - Retry when possible in AdminClient.listConsumerGroups
- [KAFKA-7379] - send.buffer.bytes should be allowed to set -1 in KafkaStreams
- [KAFKA-7386] - Streams Scala wrapper should not cache serdes
- [KAFKA-7388] - An equal sign in a property value causes the broker to fail
- [KAFKA-7400] - Compacted topic segments that precede the log start offset are not cleaned up
- [KAFKA-7403] - Offset commit failure after upgrading brokers past KIP-211/KAFKA-4682
- [KAFKA-7409] - Validate topic configs prior to topic creation
- [KAFKA-7415] - OffsetsForLeaderEpoch may incorrectly respond with undefined epoch causing truncation to HW
- [KAFKA-7434] - DeadLetterQueueReporter throws NPE if transform throws NPE
- [KAFKA-7453] - Enable idle expiry of connections which are never selected
- [KAFKA-7459] - Concurrency bug in updating RequestsPerSec metric
- [KAFKA-7460] - Connect Values converter uses incorrect date format string
- [KAFKA-7462] - Kafka brokers cannot provide OAuth without a token
- [KAFKA-7464] - Fail to shutdown ReplicaManager during broker cleaned shutdown
- [KAFKA-7467] - NoSuchElementException is raised because controlBatch is empty
- [KAFKA-7476] - SchemaProjector is not properly handling Date-based logical types
- [KAFKA-7482] - LeaderAndIsrRequest should be sent to the shutting down broker
- [KAFKA-7483] - Streams should allow headers to be passed to Serializer
- [KAFKA-7484] - Fix test SuppressionDurabilityIntegrationTest.shouldRecoverBufferAfterShutdown()
- [KAFKA-7485] - Flaky test `DyanamicBrokerReconfigurationTest.testTrustStoreAlter`
- [KAFKA-7496] - KafkaAdminClient#describeAcls should handle invalid filters gracefully
- [KAFKA-7498] - common.requests.CreatePartitionsRequest uses clients.admin.NewPartitions
- [KAFKA-7501] - double deallocation of producer batch upon expiration of inflight requests and error response
- [KAFKA-7505] - Flaky test: SslTransportLayerTest.testListenerConfigOverride
- [KAFKA-7513] - Flaky test SaslAuthenticatorFailureDelayTest.testInvalidPasswordSaslPlain
- [KAFKA-7519] - Transactional Ids Left in Pending State by TransactionStateManager During Transactional Id Expiration Are Unusable
- [KAFKA-7521] - [kafka-streams-scala_2.11] Foreach results in StackOverflowError
- [KAFKA-7535] - KafkaConsumer doesn't report records-lag if isolation.level is read_committed
- [KAFKA-7552] - StatefulProcessorNode tries to connect state store to processor before it is added
- [KAFKA-7560] - PushHttpMetricsReporter should not convert metric value to double
- [KAFKA-7561] - Console Consumer - system test fails
- [KAFKA-7579] - System Test Failure - security_test.SecurityTest.test_client_ssl_endpoint_validation_failure
Task
- [KAFKA-5887] - Replace findBugs with spotBugs for Java 9+ support
- [KAFKA-6835] - Enable topic unclean leader election to be enabled without controller change
- [KAFKA-7001] - Rename errors.allowed.max property in Connect to errors.tolerance
- [KAFKA-7002] - Allow replication factor to be set via a configuration property for the Connect DLQ topic
- [KAFKA-7003] - Add headers with error context in messages written to the Connect DeadLetterQueue topic
- [KAFKA-7139] - Support to exclude the internal topics in kafka-topics.sh command
- [KAFKA-7177] - Update 2.0 documentation to reflect changed quota behaviors by KIP-219
- [KAFKA-7228] - DeadLetterQueue throws a NullPointerException
Sub-task
- [KAFKA-6082] - consider fencing zookeeper updates with controller epoch zkVersion
- [KAFKA-7299] - batch LeaderAndIsr requests during auto preferred leader election
- [KAFKA-7308] - Fix rat and checkstyle plugins configuration for Java 11 support
- [KAFKA-7309] - Upgrade Jacoco for Java 11 support
- [KAFKA-7310] - Fix SSL tests when running with Java 11
- [KAFKA-7338] - Fix SASL Kerberos tests with Java 11
- [KAFKA-7372] - Upgrade Jetty for preliminary Java 11 and TLS 1.3 support