Main interface for consumer
Usage: Once ConsumerFetcherManager is created, startConnections() and stopAllConnections() can be called repeatedly until shutdown() is called.
An iterator that blocks until a value can be read from the supplied queue.
Tracks metrics for each topic the given consumer client has consumed data from.
Tracks metrics of the requests made by a given consumer client to all brokers, and the responses obtained from the brokers.
Range partitioning works on a per-topic basis.
Range partitioning works on a per-topic basis. For each topic, we lay out the available partitions in numeric order and the consumer threads in lexicographic order. We then divide the number of partitions by the total number of consumer streams (threads) to determine the number of partitions to assign to each consumer. If it does not evenly divide, then the first few consumers will have one extra partition. For example, suppose there are two consumers C1 and C2 with two streams each, and there are five available partitions (p0, p1, p2, p3, p4). So each consumer thread will get at least one partition and the first consumer thread will get one extra partition. So the assignment will be: p0 -> C1-0, p1 -> C1-0, p2 -> C1-1, p3 -> C2-0, p4 -> C2-1
The round-robin partition assignor lays out all the available partitions and all the available consumer threads.
The round-robin partition assignor lays out all the available partitions and all the available consumer threads. It then proceeds to do a round-robin assignment from partition to consumer thread. If the subscriptions of all consumer instances are identical, then the partitions will be uniformly distributed. (i.e., the partition ownership counts will be within a delta of exactly one across all consumer threads.)
(For simplicity of implementation) the assignor is allowed to assign a given topic-partition to any consumer instance and thread-id within that instance. Therefore, round-robin assignment is allowed only if: a) Every topic has the same number of streams within a consumer instance b) The set of subscribed topics is identical for every consumer instance within the group.
A consumer of kafka messages
A consumer of kafka messages
Stores the topic stats information of each consumer client in a (clientId -> ConsumerTopicStats) map.
Stores the fetch request and response stats information of each consumer client in a (clientId -> FetchRequestAndResponseStats) map.
An iterator that blocks until a value can be read from the supplied queue. The iterator takes a shutdownCommand object which can be added to the queue to trigger a shutdown