Release Notes - Kafka - Version 1.0.0

Below is a summary of the JIRA issues addressed in the 1.0.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at http://kafka.apache.org/documentation.html.

New Feature

[KAFKA-4602] - KIP-72 Allow putting a bound on memory consumed by Incoming requests
[KAFKA-4819] - Expose states of active tasks to public API
[KAFKA-5341] - Add UnderMinIsrPartitionCount and per-partition UnderMinIsr metrics
[KAFKA-5651] - KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines
[KAFKA-5694] - Add ChangeReplicaDirRequest and DescribeReplicaDirRequest (KIP-113)
[KAFKA-5738] - Add cumulative count attribute for all Kafka rate metrics
[KAFKA-5746] - Add new metrics to support health checks
[KAFKA-5854] - Handle SASL authentication failures as non-retriable exceptions in clients

Improvement

[KAFKA-1595] - Remove deprecated and slower scala JSON parser
[KAFKA-2376] - Add Kafka Connect metrics
[KAFKA-3741] - Allow setting of default topic configs via StreamsConfig
[KAFKA-4108] - Improve DumpLogSegments offsets-decoder output format
[KAFKA-4260] - Improve documentation of configuration listeners=PLAINTEXT://0.0.0.0:9092
[KAFKA-4322] - StateRestoreCallback begin and end indication
[KAFKA-4380] - Update the usage description of clean shutdown file.
[KAFKA-4416] - Add a '--group' option to the console consumer
[KAFKA-4501] - Support Java 9
[KAFKA-4504] - Details of retention.bytes property at Topic level are not clear on how they impact partition size
[KAFKA-4763] - Handle disk failure for JBOD (KIP-112)
[KAFKA-4764] - Improve diagnostics for SASL authentication failures
[KAFKA-5076] - Remove usage of java.xml.bind.* classes hidden by default in JDK9
[KAFKA-5077] - Make server start script work against Java 9
[KAFKA-5096] - Only log invalid user configs and overwrite with correct one
[KAFKA-5233] - Changes to punctuate semantics (KIP-138)
[KAFKA-5330] - Use per-task converters in Connect
[KAFKA-5358] - Consumer perf tool should count rebalance time separately
[KAFKA-5363] - Add ability to batch restore and receive restoration stats.
[KAFKA-5384] - KIP-162: Enable topic deletion by default
[KAFKA-5410] - Fix taskClass() method name in Connector and flush() signature in SinkTask
[KAFKA-5439] - Add checks in integration tests to verify that threads have been shutdown
[KAFKA-5454] - Add a new Kafka Streams example IoT oriented
[KAFKA-5461] - KIP-168: Add GlobalTopicCount metric per cluster
[KAFKA-5468] - Change Source offset commit message to info to match Sink behaviour
[KAFKA-5470] - Replace -XX:+DisableExplicitGC with -XX:+ExplicitGCInvokesConcurrent in kafka-run-class
[KAFKA-5485] - Streams should not suspend tasks twice
[KAFKA-5495] - Replace the deprecated 'ConsumerOffsetChecker' in documentation
[KAFKA-5515] - Consider removing date formatting from Segments class
[KAFKA-5516] - Formatting verifiable producer/consumer output in a similar fashion
[KAFKA-5525] - Streams reset tool should have same console output with or without dry-run
[KAFKA-5542] - Improve Java doc for LeaderEpochFileCache.endOffsetFor()
[KAFKA-5547] - Return topic authorization failed if no topic describe access
[KAFKA-5549] - Explain that `client.id` is just used as a prefix within Streams
[KAFKA-5557] - Using a logPrefix inside the StreamPartitionAssignor
[KAFKA-5602] - Add --custom-ducktape flag to ducker-ak
[KAFKA-5629] - Console Consumer overrides auto.offset.reset property when provided on the command line without warning about it.
[KAFKA-5657] - Connect REST API should include the connector type when describing a connector
[KAFKA-5695] - Test DeleteRecordsRequest in AuthorizerIntegrationTest
[KAFKA-5726] - KafkaConsumer.subscribe() overload that takes just Pattern without ConsumerRebalanceListener
[KAFKA-5743] - All ducktape services should store their files in subdirectories of /mnt
[KAFKA-5744] - ShellTest: add tests for attempting to run nonexistent program, error return
[KAFKA-5750] - Elevate log messages for denials to INFO in SimpleAclAuthorizer class
[KAFKA-5754] - Refactor Streams to use LogContext
[KAFKA-5755] - Refactor Producer to use LogContext
[KAFKA-5762] - Refactor AdminClient to use LogContext
[KAFKA-5763] - Refactor NetworkClient to use LogContext
[KAFKA-5768] - Upgrade ducktape version to 0.7.1, and use new kill_java_processes
[KAFKA-5783] - Implement KafkaPrincipalBuilder interface with support for SASL (KIP-189)
[KAFKA-5820] - Remove unneeded synchronized keyword in StreamThread
[KAFKA-5822] - Consistent logging of topic partitions
[KAFKA-5856] - Add AdminClient.createPartitions()
[KAFKA-5915] - Support unmapping of mapped/direct buffers in Java 9
[KAFKA-5920] - Handle SSL authentication failures as non-retriable exceptions in clients
[KAFKA-5932] - Avoid call fetchPrevious in FlushListeners
[KAFKA-5947] - Handle authentication failures from transactional producer and KafkaAdminClient
[KAFKA-5958] - User StoreListener not available for global stores
[KAFKA-5985] - Mention the need to close store iterators
[KAFKA-6004] - Enable custom authentication plugins to return error messages to clients
[KAFKA-6025] - There is something wrong in Kafka Streams document

Bug

[KAFKA-2105] - NullPointerException in client on MetadataRequest
[KAFKA-2360] - The kafka-consumer-perf-test.sh script help information print useless parameters.
[KAFKA-3356] - Remove ConsumerOffsetChecker, deprecated in 0.9, in 0.11
[KAFKA-3623] - Make KStreamTestDriver extending from ExternalResource
[KAFKA-3856] - Cleanup Kafka Streams builder API
[KAFKA-3986] - completedReceives can contain closed channels
[KAFKA-4585] - KIP-163: Offset fetch and commit requests use the same permissions
[KAFKA-4669] - KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception
[KAFKA-4785] - Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor
[KAFKA-4830] - Augment KStream.print() to allow users pass in extra parameters in the printed string
[KAFKA-4856] - Calling KafkaProducer.close() from multiple threads may cause spurious error
[KAFKA-4860] - Kafka batch files does not support path with spaces
[KAFKA-4913] - creating a window store with one segment throws division by zero error
[KAFKA-5127] - Replace pattern matching with foreach where the case None is unused
[KAFKA-5140] - Flaky ResetIntegrationTest
[KAFKA-5152] - Kafka Streams keeps restoring state after shutdown is initiated during startup
[KAFKA-5167] - streams task gets stuck after re-balance due to LockException
[KAFKA-5225] - StreamsResetter doesn't allow custom Consumer properties
[KAFKA-5239] - Producer buffer pool allocates memory inside a lock.
[KAFKA-5379] - ProcessorContext.appConfigs() should return parsed/validated values
[KAFKA-5402] - JmxReporter Fetch metrics for kafka.server should not be created when client quotas are not enabled
[KAFKA-5412] - Using connect-console-sink/source.properties raises an exception related to "file" property not found
[KAFKA-5417] - Clients get inconsistent connection states when SASL/SSL connection is marked CONECTED and DISCONNECTED at the same time
[KAFKA-5431] - LogCleaner stopped due to org.apache.kafka.common.errors.CorruptRecordException
[KAFKA-5464] - StreamsKafkaClient should not use StreamsConfig.POLL_MS_CONFIG
[KAFKA-5469] - Created state changelog topics not logged correctly
[KAFKA-5481] - ListOffsetResponse isn't logged in the right way with trace level enabled
[KAFKA-5484] - Refactor kafkatest docker support
[KAFKA-5487] - Rolling upgrade test for streams
[KAFKA-5506] - bin/kafka-consumer-groups.sh failing to query offsets
[KAFKA-5507] - kafka-run-class.sh doesn't check for empty $CLASSPATH
[KAFKA-5508] - Documentation for altering topics
[KAFKA-5512] - KafkaConsumer: High memory allocation rate when idle
[KAFKA-5534] - KafkaConsumer offsetsForTimes should include partitions in result even if no offset could be found
[KAFKA-5535] - Transformations - tranformations for value broken on tombstone events
[KAFKA-5544] - The LastStableOffsetLag metric should be removed when partition is deleted
[KAFKA-5548] - SchemaBuilder does not validate input.
[KAFKA-5556] - KafkaConsumer.commitSync throws IllegalStateException: Attempt to retrieve exception from future which hasn't failed
[KAFKA-5566] - Instable test QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied
[KAFKA-5567] - With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition
[KAFKA-5576] - Support Power platform by updating rocksdb version
[KAFKA-5579] - SchemaBuilder.type(Schema.Type) should not allow null.
[KAFKA-5587] - Processor got uncaught exception: NullPointerException
[KAFKA-5595] - Illegal state in SocketServer; attempt to send with another send in progress
[KAFKA-5599] - ConsoleConsumer : --new-consumer option as deprecated
[KAFKA-5600] - Group loading regression causing stale metadata/offsets cache
[KAFKA-5603] - Streams should not abort transaction when closing zombie task
[KAFKA-5607] - Improve error handling in SocketServer to reduce illegal states later
[KAFKA-5608] - System test failure due to timeout starting Jmx tool
[KAFKA-5610] - KafkaApis.handleWriteTxnMarkerRequest can return UNSUPPORTED_FOR_MESSAGE_FORMAT error on partition emigration
[KAFKA-5619] - Make --new-consumer option as deprecated in all tools
[KAFKA-5620] - SerializationException in doSend() masks class cast exception
[KAFKA-5623] - ducktape kafka service: do not assume Service contains num_nodes
[KAFKA-5630] - Consumer poll loop over the same record after a CorruptRecordException
[KAFKA-5643] - Using _DUCKTAPE_OPTIONS has no effect on executing tests
[KAFKA-5644] - Transient test failure: ResetConsumerGroupOffsetTest.testResetOffsetsToZonedDateTime
[KAFKA-5656] - Support bulk attributes request on KafkaMbean where some attributes do not exist
[KAFKA-5663] - LogDirFailureTest system test fails
[KAFKA-5668] - queryable state window store range scan only returns results from one store
[KAFKA-5679] - Add logging to distinguish between internally and externally initiated shutdown of Kafka
[KAFKA-5698] - Sort processor node based on name suffix in TopologyDescription.toString()
[KAFKA-5701] - Unit test shouldTogglePrepareForBulkLoadDuringRestoreCalls fails
[KAFKA-5704] - Auto topic creation causes failure with older clusters
[KAFKA-5711] - Bulk Restore Should Handle Deletes
[KAFKA-5717] - [streams] 'null' values in state stores
[KAFKA-5720] - In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with org.apache.kafka.common.errors.TimeoutException
[KAFKA-5730] - Consumer should ensure consistent ordering between sync and async offset commits
[KAFKA-5733] - System tests get exception RocksDBException: db has more levels than options.num_levels
[KAFKA-5735] - Client-ids are not handled consistently by clients and broker
[KAFKA-5737] - KafkaAdminClient thread should be daemon
[KAFKA-5745] - Partition.makeLeader() should convert HW to OffsetMetadata before becoming the leader
[KAFKA-5752] - Delete topic and re-create topic immediate will delete the new topic's timeindex
[KAFKA-5753] - ShellTest.testRunProgramWithErrorReturn fails on macOS
[KAFKA-5756] - Synchronization issue on flush
[KAFKA-5758] - Reassigning a topic's partitions can adversely impact other topics
[KAFKA-5765] - Move merge() from StreamsBuilder to KStream
[KAFKA-5767] - Kafka server should halt if IBP < 1.0.0 and there is log directory failure
[KAFKA-5771] - org.apache.kafka.streams.state.internals.Segments#segments method returns incorrect results when segments were added out of order
[KAFKA-5774] - WorkerSinkTask failure exception should include cause
[KAFKA-5787] - StoreChangeLogReader needs to restore partitions that were added post initialization
[KAFKA-5790] - SocketServer.processNewResponses should not skip a response if exception is thrown
[KAFKA-5797] - StoreChangelogReader should be resilient to broker-side metadata not available
[KAFKA-5804] - ChangeLoggingWindowBytesStore needs to retain duplicates when writing to the log
[KAFKA-5818] - KafkaStreams state transitions not correct
[KAFKA-5829] - Speedup broker startup after unclean shutdown by reducing unnecessary snapshot files deletion
[KAFKA-5833] - Reset thread interrupt state in case of InterruptedException
[KAFKA-5837] - ReassignPartitionsCommand fails if default throttle/timeout used
[KAFKA-5843] - Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to true
[KAFKA-5862] - Remove ZK dependency from Streams reset tool
[KAFKA-5864] - ReplicaFetcherThread should not die due to replica in offline log directory
[KAFKA-5865] - Expiring batches with idempotence enabled could cause data loss.
[KAFKA-5867] - Kafka Connect applications should log info message when starting up
[KAFKA-5872] - Transient failure in SslSelectorTest.testMuteOnOOM
[KAFKA-5879] - Controller should read the latest IsrChangeNotification znodes when handling IsrChangeNotification event
[KAFKA-5908] - CompositeReadOnlyWindowStore range fetch doesn't return all values when fetching with different start and end times
[KAFKA-5918] - Fix minor typos and errors in the Kafka Streams turotial
[KAFKA-5953] - Connect classloader isolation may be broken for JDBC drivers
[KAFKA-5954] - Failure in Connect system test: ConnectRestApiTest
[KAFKA-5956] - StreamBuilder#table and StreamsBuilder#globalTable should use serdes from Materialized
[KAFKA-5957] - Producer IllegalStateException due to second deallocate after aborting a batch
[KAFKA-5959] - NPE in NetworkClient
[KAFKA-5960] - Producer uses unsupported ProduceRequest version against older brokers
[KAFKA-5967] - Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()
[KAFKA-5968] - Remove all broker metrics during shutdown
[KAFKA-5970] - Deadlock due to locking of DelayedProduce and group
[KAFKA-5976] - RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE logging
[KAFKA-5979] - kip-120 changed internal streams topic naming
[KAFKA-5980] - FailOnInvalidTimestamp does not log error
[KAFKA-5986] - Streams State Restoration never completes when logging is disabled
[KAFKA-5989] - disableLogging() causes partitions to not be consumed
[KAFKA-5995] - Rename AlterReplicaDir to AlterReplicaDirs
[KAFKA-6003] - Replication Fetcher thread for a partition with no data fails to start
[KAFKA-6005] - Reject JoinGroup request from first member with empty protocol type/protocol list
[KAFKA-6008] - Kafka Connect: Unsanitized workerID causes exception during startup
[KAFKA-6010] - Transient failure: MemoryRecordsBuilderTest.convertToV1WithMixedV0AndV2Data
[KAFKA-6012] - NoSuchElementException in markErrorMeter during TransactionsBounceTest
[KAFKA-6015] - NPE in RecordAccumulator
[KAFKA-6016] - Use the idempotent producer in the reassign_partitions_test
[KAFKA-6026] - KafkaFuture timeout fails to fire if a narrow race condition is hit
[KAFKA-6027] - System test failure: LogDirFailureTest
[KAFKA-6030] - Integer overflow in log cleaner cleanable ratio computation
[KAFKA-6042] - Kafka Request Handler deadlocks and brings down the cluster.
[KAFKA-6053] - NoSuchMethodError when creating ProducerRecord in upgrade system tests
[KAFKA-6055] - Running tools on Windows fail due to typo in JVM config
[KAFKA-6070] - ducker-ak: add ipaddress and enum34 dependencies to docker image
[KAFKA-6087] - Scanning plugin.path needs to support relative symlinks
[KAFKA-6093] - replica dir not deleted after topic deletion
[KAFKA-6100] - Streams quick start crashes Java on Windows
[KAFKA-6101] - Reconnecting to broker does not exponentially backoff
[KAFKA-6116] - Major performance issue due to excessive logging during leader election
[KAFKA-6119] - Silent Data Loss in Kafka011 Transactional Producer
[KAFKA-6131] - Transaction markers are sometimes discarded if txns complete concurrently
[KAFKA-6134] - High memory usage on controller during partition reassignment

Task

[KAFKA-5708] - Update Jackson dependencies (from 2.8.5 to 2.9.x)
[KAFKA-5884] - Enable PowerMock tests when running on Java 9
[KAFKA-5977] - Upgrade RocksDB dependency to legally acceptable version

Test

[KAFKA-5742] - Support passing ZK chroot in system tests
[KAFKA-6104] - Add unit tests for ClusterConnectionStates

Sub-task

[KAFKA-2507] - Replace ControlledShutdown{Request,Response} with org.apache.kafka.common.requests equivalent
[KAFKA-2959] - Remove temporary mapping to deserialize functions in RequestChannel
[KAFKA-4593] - Task migration during rebalance callback process could lead the obsoleted task's IllegalStateException
[KAFKA-4643] - Improve test coverage of StreamsKafkaClient
[KAFKA-4653] - Improve test coverage of RocksDBStore
[KAFKA-4655] - Improve test coverage of CompositeReadOnlySessionStore
[KAFKA-4656] - Improve test coverage of CompositeReadOnlyKeyValueStore
[KAFKA-4658] - Improve test coverage InMemoryKeyValueLoggedStore
[KAFKA-4659] - Improve test coverage of CachingKeyValueStore
[KAFKA-4661] - Improve test coverage UsePreviousTimeOnInvalidTimestamp
[KAFKA-4868] - Optimize RocksDb config for fast recovery/bulk load
[KAFKA-5006] - KeyValueStore.put may throw exception unrelated to the current put attempt
[KAFKA-5134] - Replace zkClient.getChildren method with zkUtils.getChildren
[KAFKA-5157] - Options for handling corrupt data during deserialization
[KAFKA-5314] - Improve exception handling for state stores
[KAFKA-5342] - Distinguish abortable failures in transactional producer
[KAFKA-5372] - Unexpected state transition Dead to PendingShutdown
[KAFKA-5388] - Replace zkClient.subscribe*Changes method with an equivalent zkUtils method
[KAFKA-5389] - Replace zkClient.exists method with zkUtils.pathExists
[KAFKA-5391] - Replace zkClient.delete* method with an equivalent zkUtils method
[KAFKA-5494] - Idempotent producer should not require max.in.flight.requests.per.connection=1
[KAFKA-5501] - introduce async ZookeeperClient
[KAFKA-5531] - Check exception handling in all streams tests
[KAFKA-5597] - Autogenerate Producer sender metrics
[KAFKA-5650] - Provide a simple way for custom storage engines to use streams wrapped stores (KIP-182)
[KAFKA-5653] - Add new API methods to KTable
[KAFKA-5654] - Add new API methods to KGroupedStream
[KAFKA-5655] - Add new API methods to KGroupedTable
[KAFKA-5670] - Add Topology and deprecate TopologyBuilder
[KAFKA-5671] - Add StreamsBuilder and deprecate KStreamBuilder
[KAFKA-5673] - Refactor KeyValueStore hierarchy so that MeteredKeyValueStore is the outermost store
[KAFKA-5689] - Refactor WindowStore hierarchy so that Metered Store is the outermost store
[KAFKA-5727] - Add the archetype project along with "write applications" web docs.
[KAFKA-5749] - Refactor SessionStore hierarchy
[KAFKA-5776] - Add the Trogdor fault injection daemon
[KAFKA-5777] - Add ducktape integration for the Trogdor Fault injection daemon
[KAFKA-5793] - Tighten up situations where OutOfOrderSequence may be returned
[KAFKA-5806] - Fix transient unit test failure in trogdor coordinator shutdown
[KAFKA-5815] - Add Printed class and KStream#print(Printed)
[KAFKA-5816] - Add Produced class and new to and through overloads to KStream
[KAFKA-5817] - Add Serialized class and KStream groupBy and groupByKey overloads
[KAFKA-5819] - Add Joined class and relevant KStream join overloads
[KAFKA-5823] - Update Docs
[KAFKA-5832] - Add Consumed class and overloads to StreamBuilder
[KAFKA-5839] - Upgrade Guide doc changes for KIP-130
[KAFKA-5844] - Add groupBy(KeyValueMapper, Serialized) to KTable
[KAFKA-5852] - Add filter, filterNot, mapValues and Materialized to KTable
[KAFKA-5853] - Add WindowedKStream interface and implemenation
[KAFKA-5873] - Add Materialized overloads to StreamBuilder
[KAFKA-5888] - Transactions system test should check for message order
[KAFKA-5899] - Create Connect metrics for connectors
[KAFKA-5900] - Create Connect metrics common to source and sink tasks
[KAFKA-5901] - Create Connect metrics for source tasks
[KAFKA-5902] - Create Connect metrics for sink tasks
[KAFKA-5903] - Create Connect metrics for workers
[KAFKA-5904] - Create Connect metrics for worker rebalances
[KAFKA-5913] - Add `hasOffset` and `hasTimestamp` methods to RecordMetadata to indicate whether metadata is available
[KAFKA-5921] - Add Materialized overloads to WindowedKStream
[KAFKA-5922] - Add SessionWindowedKStream
[KAFKA-5931] - Deprecate KTable#to and KTable#through
[KAFKA-5937] - Improve ProcessorStateManager exception handling
[KAFKA-5949] - User Callback Exceptions need to be handled properly
[KAFKA-5990] - Add generated documentation for Connect metrics