Release Notes - Kafka - Version 1.1.0

Below is a summary of the JIRA issues addressed in the 1.1.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at http://kafka.apache.org/documentation.html.

New Feature

• [KAFKA-1689] - automatic migration of log dirs to new locations
• [KAFKA-5142] - KIP-145 - Expose Record Headers in Kafka Connect
• [KAFKA-6170] - Add the AdminClient in Streams' KafkaClientSupplier
• [KAFKA-6240] - Support dynamic updates of frequently updated broker configs

Improvement

• [KAFKA-3073] - KafkaConnect should support regular expression for topics
• [KAFKA-3625] - Move kafka-streams test fixtures into a published package
• [KAFKA-4029] - SSL support for Connect REST API
• [KAFKA-4115] - Grow default heap settings for distributed Connect from 256M to 1G
• [KAFKA-4218] - Enable access to key in ValueTransformer
• [KAFKA-4499] - Add "getAllKeys" API for querying windowed KTable stores
• [KAFKA-4706] - Unify StreamsKafkaClient instances
• [KAFKA-5228] - Revisit Streams DSL JavaDocs
• [KAFKA-5520] - Extend Consumer Group Reset Offset tool for Stream Applications
• [KAFKA-5526] - KIP-175: ConsumerGroupCommand no longer shows output for consumer groups which have not committed offsets
• [KAFKA-5565] - Add a broker metric specifying the number of consumer group rebalances in progress
• [KAFKA-5631] - Use Jackson for serialising to JSON
• [KAFKA-5764] - KafkaShortnamer should allow for case-insensitive matches
• [KAFKA-5772] - Improve Util classes
• [KAFKA-5895] - Gradle 3.0+ is needed on the build
• [KAFKA-5925] - Adding records deletion operation to the new Admin Client API
• [KAFKA-5987] - Kafka metrics templates used in document generation should maintain order of tags
• [KAFKA-6086] - Provide for custom error handling when Kafka Streams fails to produce
• [KAFKA-6150] - Make Repartition Topics Transient
• [KAFKA-6168] - Connect Schema comparison is slow for large schemas
• [KAFKA-6180] - Add a Validator for NonNull configurations and remove redundant null checks on lists
• [KAFKA-6205] - Have State Stores Restore Before Initializing Toplogy
• [KAFKA-6218] - Optimize condition in if statement to reduce the number of comparisons
• [KAFKA-6233] - Removed unnecessary null check
• [KAFKA-6254] - Introduce Incremental FetchRequests to Increase Partition Scalability
• [KAFKA-6258] - SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data
• [KAFKA-6274] - Improve KTable Source state store auto-generated names
• [KAFKA-6275] - Extend consumer offset reset tool to support deletion (KIP-229)
• [KAFKA-6311] - Expose Kafka cluster ID in Connect REST API
• [KAFKA-6317] - Maven artifact for kafka should not depend on log4j
• [KAFKA-6321] - ConsumerGroupCommand should use the new consumer to query the log end offsets.
• [KAFKA-6324] - Change LogSegment.delete to deleteIfExists and harden log recovery
• [KAFKA-6382] - Make ProducerConfig and ConsumerConfig constructors public
• [KAFKA-6412] - Improve synchronization in CachingKeyValueStore methods
• [KAFKA-6430] - Improve Kafka GZip compression performance
• [KAFKA-6456] - Improve JavaDoc of SourceTask#poll() to discourage indefinite blocking
• [KAFKA-6499] - Avoid creating dummy checkpoint files with no state stores
• [KAFKA-6519] - Change log level from ERROR to WARN for not leader for this partition exception
• [KAFKA-6634] - Delay initiating the txn on producers until initializeTopology with EOS turned on

Bug

• [KAFKA-1044] - change log4j to slf4j
• [KAFKA-2394] - Use RollingFileAppender by default in log4j.properties
• [KAFKA-2729] - Cached zkVersion not equal to that in zookeeper, broker not recovering.
• [KAFKA-2903] - FileRecords.read doesn't handle size > sizeInBytes when start is not zero
• [KAFKA-3083] - a soft failure in controller may leave a topic partition in an inconsistent state
• [KAFKA-3978] - Cannot truncate to a negative offset (-1) exception at broker startup
• [KAFKA-4084] - automated leader rebalance causes replication downtime for clusters with too many partitions
• [KAFKA-4247] - kafka-run-class has potential to add a leading colon to classpath
• [KAFKA-4263] - QueryableStateIntegrationTest.concurrentAccess is failing occasionally in jenkins builds
• [KAFKA-4335] - FileStreamSource Connector not working for large files (~ 1GB)
• [KAFKA-4750] - KeyValueIterator returns null values
• [KAFKA-4767] - KafkaProducer is not joining its IO thread properly
• [KAFKA-4827] - Kafka connect: error with special characters in connector name
• [KAFKA-4857] - Replace StreamsKafkaClient with AdminClient in Kafka Streams
• [KAFKA-4897] - LogCleaner#cleanSegments should not ignore failures to delete files
• [KAFKA-4930] - Connect Rest API allows creating connectors with an empty name - KIP-212
• [KAFKA-4991] - KerberosLogin#login should probably be synchronized
• [KAFKA-5163] - Support replicas movement between log directories (KIP-113)
• [KAFKA-5212] - Consumer ListOffsets request can starve group heartbeats
• [KAFKA-5550] - Struct.put() should include the field name if validation fails
• [KAFKA-5563] - Clarify handling of connector name in config
• [KAFKA-5759] - Allow user to specify relative path as log directory
• [KAFKA-5848] - KafkaConsumer should validate topics/TopicPartitions on subscribe/assign
• [KAFKA-5877] - Controller should only update reassignment znode if there is change in the reassignment data
• [KAFKA-5890] - records.lag should use tags for topic and partition rather than using metric name.
• [KAFKA-5936] - KafkaProducer should not wrap InterruptedException in close() with KafkaException
• [KAFKA-5953] - Connect classloader isolation may be broken for JDBC drivers
• [KAFKA-5996] - JsonConverter generates "Mismatching schema" DataException
• [KAFKA-6012] - NoSuchElementException in markErrorMeter during TransactionsBounceTest
• [KAFKA-6018] - Make KafkaFuture.Function java 8 lambda compatible
• [KAFKA-6023] - ThreadCache#sizeBytes() should check overflow
• [KAFKA-6026] - KafkaFuture timeout fails to fire if a narrow race condition is hit
• [KAFKA-6046] - DeleteRecordsRequest to a non-leader
• [KAFKA-6051] - ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown
• [KAFKA-6053] - NoSuchMethodError when creating ProducerRecord in upgrade system tests
• [KAFKA-6055] - Running tools on Windows fail due to typo in JVM config
• [KAFKA-6069] - Streams metrics tagged incorrectly
• [KAFKA-6070] - ducker-ak: add ipaddress and enum34 dependencies to docker image
• [KAFKA-6075] - Kafka cannot recover after an unclean shutdown on Windows
• [KAFKA-6087] - Scanning plugin.path needs to support relative symlinks
• [KAFKA-6101] - Reconnecting to broker does not exponentially backoff
• [KAFKA-6115] - TaskManager should be type aware
• [KAFKA-6116] - Major performance issue due to excessive logging during leader election
• [KAFKA-6120] - RecordCollectorImpl should not retry sending
• [KAFKA-6121] - Restore and global consumer should not use auto.offset.reset
• [KAFKA-6122] - Global Consumer should handle TimeoutException
• [KAFKA-6126] - Reduce rebalance time by not checking if created topics are available
• [KAFKA-6130] - VerifiableConsumer with --max-messages doesn't exit
• [KAFKA-6131] - Transaction markers are sometimes discarded if txns complete concurrently
• [KAFKA-6138] - Simplify StreamsBuilder#addGlobalStore
• [KAFKA-6156] - JmxReporter can't handle windows style directory paths
• [KAFKA-6158] - CONSUMER-ID and HOST values are concatenated if the CONSUMER-ID is > 50 chars
• [KAFKA-6164] - ClientQuotaManager threads prevent shutdown when encountering an error loading logs
• [KAFKA-6166] - Streams configuration requires consumer. and producer. in order to be read
• [KAFKA-6167] - Timestamp on streams directory contains a colon, which is an illegal character
• [KAFKA-6174] - Add methods in Options classes to keep binary compatibility with 0.11
• [KAFKA-6179] - RecordQueue.clear() does not clear MinTimestampTracker's maintained list
• [KAFKA-6185] - Selector memory leak with high likelihood of OOM in case of down conversion
• [KAFKA-6190] - GlobalKTable never finishes restoring when consuming transactional messages
• [KAFKA-6194] - Server crash while deleting segments
• [KAFKA-6210] - IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version
• [KAFKA-6214] - Using standby replicas with an in memory state store causes Streams to crash
• [KAFKA-6238] - Issues with protocol version when applying a rolling upgrade to 1.0.0
• [KAFKA-6247] - Fix system test dependency issues
• [KAFKA-6250] - Kafka Connect requires permission to create internal topics even if they exist
• [KAFKA-6252] - A metric named 'XX' already exists, can't register another one.
• [KAFKA-6253] - Improve sink connector topic regex validation
• [KAFKA-6256] - Flaky Unit test: KStreamKTableJoinIntegrationTest.shouldCountClicksPerRegionWithNonZeroByteCache
• [KAFKA-6259] - Make KafkaStreams.cleanup() clean global state directory
• [KAFKA-6260] - AbstractCoordinator not clearly handles NULL Exception
• [KAFKA-6261] - Request logging throws exception if acks=0
• [KAFKA-6265] - GlobalKTable missing #queryableStoreName()
• [KAFKA-6269] - KTable state restore fails after rebalance
• [KAFKA-6277] - Make loadClass thread-safe for class loaders of Connect plugins
• [KAFKA-6284] - System Test failed: ConnectRestApiTest
• [KAFKA-6288] - Broken symlink interrupts scanning the plugin path
• [KAFKA-6289] - NetworkClient should not return internal failed api version responses from poll
• [KAFKA-6296] - Transient failure in NetworkClientTest.testConnectionDelayDisconnected
• [KAFKA-6298] - Line numbers on log messages are incorrect
• [KAFKA-6302] - Topic can not be recreated after it is deleted
• [KAFKA-6307] - mBeanName should be removed before returning from JmxReporter#removeAttribute()
• [KAFKA-6313] - Kafka Core should have explicit SLF4J API dependency
• [KAFKA-6318] - StreamsResetter should return non-zero return code on error
• [KAFKA-6319] - kafka-acls regression for comma characters (and maybe other characters as well)
• [KAFKA-6322] - Error deleting log for topic, all log dirs failed.
• [KAFKA-6323] - punctuate with WALL_CLOCK_TIME triggered immediately
• [KAFKA-6328] - Exclude node groups belonging to global stores in InternalTopologyBuilder#makeNodeGroups
• [KAFKA-6331] - Transient failure in kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpointkafka.api.AdminClientIntegrationTest.testAlterReplicaLogDirs
• [KAFKA-6345] - NetworkClient.inFlightRequestCount() is not thread safe, causing ConcurrentModificationExceptions when sensors are read
• [KAFKA-6349] - ConcurrentModificationException during streams state restoration
• [KAFKA-6354] - Update KStream JavaDoc with regard to KIP-182
• [KAFKA-6360] - RocksDB segments not removed when store is closed causes re-initialization to fail
• [KAFKA-6362] - auto commit not work since coordinatorUnknown() is always true.
• [KAFKA-6367] - Fix StateRestoreListener To Use Correct Batch Ending Offset
• [KAFKA-6378] - NullPointerException on KStream-GlobalKTable leftJoin when KeyValueMapper returns null
• [KAFKA-6383] - StreamThread.shutdown doesn't clean up completely when called before StreamThread.start
• [KAFKA-6390] - Update ZooKeeper to 3.4.11, Gradle and other minor updates
• [KAFKA-6391] - output from ensure copartitioning is not used for Cluster metadata, resulting in partitions without tasks working on them
• [KAFKA-6397] - Consumer should not block setting initial positions of unavailable partitions
• [KAFKA-6398] - Non-aggregation KTable generation operator does not construct value getter correctly
• [KAFKA-6418] - AdminClient should handle empty or null topic names better
• [KAFKA-6422] - When enable trace level log in mirror maker, it will throw null pointer exception and the mirror maker will shutdown
• [KAFKA-6451] - Simplify KStreamReduce
• [KAFKA-6464] - Base64URL encoding under JRE 1.7 is broken due to incorrect padding assumption
• [KAFKA-6487] - ChangeLoggingKeyValueBytesStore.all() returns null
• [KAFKA-6489] - Fetcher.retrieveOffsetsByTimes() should add all the topics to the metadata refresh topics set.
• [KAFKA-6492] - LogSemgent.truncateTo() should always resize the index file
• [KAFKA-6503] - Connect: Plugin scan is very slow
• [KAFKA-6504] - Connect: Some per-task-metrics not working
• [KAFKA-6511] - Connect header parser incorrectly parses arrays
• [KAFKA-6512] - Java Producer: Excessive memory usage with compression enabled
• [KAFKA-6513] - New Connect header support doesn't define `converter.type` property correctly
• [KAFKA-6517] - ZooKeeperClient holds a lock while waiting for responses, blocking shutdown
• [KAFKA-6528] - Transient failure in DynamicBrokerReconfigurationTest.testThreadPoolResize
• [KAFKA-6529] - Broker leaks memory and file descriptors after sudden client disconnects
• [KAFKA-6532] - Delegation token internals should not impact public interfaces
• [KAFKA-6536] - Streams quickstart pom.xml is missing versions for a bunch of plugins
• [KAFKA-6549] - Deadlock while processing Controller Events
• [KAFKA-6554] - Broker doesn't reject Produce request with inconsistent state
• [KAFKA-6568] - LogCleanerManager.doneDeleting() should check the partition state before deleting the in progress partition
• [KAFKA-6573] - KafkaController.brokerInfo not updated on dynamic update
• [KAFKA-6577] - Connect standalone SASL file source and sink test fails without explanation
• [KAFKA-6578] - Connect distributed and standalone worker 'main()' methods should catch and log all exceptions
• [KAFKA-6590] - Consumer bytes-fetched and records-fetched metrics are not aggregated correctly
• [KAFKA-6593] - Coordinator disconnect in heartbeat thread can cause commitSync to block indefinitely
• [KAFKA-6606] - Regression in consumer auto-commit backoff behavior
• [KAFKA-6622] - GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly
• [KAFKA-6624] - log segment deletion could cause a disk to be marked offline incorrectly
• [KAFKA-6653] - Delayed operations may not be completed when there is lock contention
• [KAFKA-6663] - Expression for GlobalKTable is not correct
• [KAFKA-6676] - System tests do not handle ZK chroot properly with SCRAM
• [KAFKA-6680] - Fix config initialization in DynamicBrokerConfig
• [KAFKA-6683] - ReplicaFetcher crashes with "Attempted to complete a transaction which was not started"

Test

• [KAFKA-6096] - Add concurrent tests to exercise all paths in group/transaction managers
• [KAFKA-6104] - Add unit tests for ClusterConnectionStates
• [KAFKA-6193] - Only delete reassign_partitions znode after reassignment is complete
• [KAFKA-6215] - KafkaStreamsTest fails in trunk
• [KAFKA-6300] - SelectorTest may fail with ConcurrentModificationException

Sub-task

• [KAFKA-4541] - Add capability to create delegation token
• [KAFKA-4542] - Add authentication based on delegation token.
• [KAFKA-4543] - Add capability to renew/expire delegation tokens.
• [KAFKA-5083] - always leave the last surviving member of the ISR in ZK
• [KAFKA-5473] - handle ZK session expiration properly when a new session can't be established
• [KAFKA-5541] - Streams should not re-throw if suspending/closing tasks fails
• [KAFKA-5642] - Use async ZookeeperClient in Controller
• [KAFKA-5645] - Use async ZookeeperClient in SimpleAclAuthorizer
• [KAFKA-5646] - Use async ZookeeperClient for DynamicConfigManager
• [KAFKA-5647] - Use async ZookeeperClient for Admin operations
• [KAFKA-5811] - Trogdor should handle injecting disk faults
• [KAFKA-5849] - Add process stop faults, round trip workload, partitioned produce-consume test
• [KAFKA-5894] - add the notion of max inflight requests to async ZookeeperClient
• [KAFKA-6060] - Add workload generation capabilities to Trogdor
• [KAFKA-6065] - Add ZooKeeperRequestLatencyMs to KafkaZkClient
• [KAFKA-6071] - Use ZookeeperClient in LogManager
• [KAFKA-6072] - Use ZookeeperClient in GroupCoordinator and TransactionCoordinator
• [KAFKA-6073] - Use ZookeeperClient in KafkaApis
• [KAFKA-6074] - Use ZookeeperClient in ReplicaManager and Partition
• [KAFKA-6111] - Tests for KafkaZkClient
• [KAFKA-6118] - Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials
• [KAFKA-6146] - minimize the number of triggers enqueuing PreferredReplicaLeaderElection events
• [KAFKA-6241] - Enable dynamic reconfiguration of SSL keystores
• [KAFKA-6242] - Enable resizing various broker thread pools
• [KAFKA-6243] - Enable reconfiguration of metrics reporters and their custom configs
• [KAFKA-6244] - Enable dynamic reconfiguration of log cleaners
• [KAFKA-6245] - Enable reconfiguration of default topic configs used by brokers
• [KAFKA-6246] - Enable reconfiguration of listeners and security configs
• [KAFKA-6255] - Add ProduceBench to Trogdor
• [KAFKA-6320] - move ZK metrics in KafkaHealthCheck to ZookeeperClient
• [KAFKA-6452] - Add documentation for delegation token authentication mechanism
• [KAFKA-6476] - Document dynamic config update
• [KAFKA-6494] - Extend ConfigCommand to update broker config using new AdminClient
• [KAFKA-6501] - Add test to verify markPartitionsForTruncation after fetcher thread pool resize