Package

kafka

server

Permalink

package server

Visibility
  1. Public
  2. All

Type Members

  1. abstract class AbstractFetcherManager extends Logging with KafkaMetricsGroup

    Permalink
  2. abstract class AbstractFetcherThread extends ShutdownableThread

    Permalink

    Abstract class for fetching data from multiple partitions from the same broker.

  3. case class BecomeLeaderOrFollowerResult(responseMap: Map[(String, Int), Short], errorCode: Short) extends Product with Serializable

    Permalink
  4. case class BrokerAndFetcherId(broker: BrokerEndPoint, fetcherId: Int) extends Product with Serializable

    Permalink
  5. case class BrokerAndInitialOffset(broker: BrokerEndPoint, initOffset: Long) extends Product with Serializable

    Permalink
  6. case class BrokerMetadata(brokerId: Int) extends Product with Serializable

    Permalink
  7. class BrokerMetadataCheckpoint extends Logging

    Permalink

    This class saves broker's metadata to a file

  8. case class BrokerState() extends Product with Serializable

    Permalink
  9. sealed trait BrokerStates extends AnyRef

    Permalink

    Broker states are the possible state that a kafka broker can be in.

    Broker states are the possible state that a kafka broker can be in. A broker should be only in one state at a time. The expected state transition with the following defined states is:

    +-----------+ |Not Running| +-----+-----+ | v +-----+-----+ |Starting +--+ +-----+-----+ | +----+------------+ | +>+RecoveringFrom | v |UncleanShutdown | +----------+ +-----+-----+ +-------+---------+ |RunningAs | |RunningAs | | |Controller+<--->+Broker +<-----------+ +----------+ +-----+-----+ | | | v | +-----+------------+ |-----> |PendingControlled | |Shutdown | +-----+------------+ | v +-----+----------+ |BrokerShutting | |Down | +-----+----------+ | v +-----+-----+ |Not Running| +-----------+

    Custom states is also allowed for cases where there are custom kafka states for different scenarios.

  10. class BrokerTopicMetrics extends KafkaMetricsGroup

    Permalink
  11. class ClientIdConfigHandler extends ConfigHandler

    Permalink

    The ClientIdConfigHandler will process clientId config changes in ZK.

    The ClientIdConfigHandler will process clientId config changes in ZK. The callback provides the clientId and the full properties set read from ZK. This implementation reports the overrides to the respective ClientQuotaManager objects

  12. case class ClientIdTopicPartition(clientId: String, topic: String, partitionId: Int) extends Product with Serializable

    Permalink
  13. class ClientQuotaManager extends Logging

    Permalink

    Helper class that records per-client metrics.

    Helper class that records per-client metrics. It is also responsible for maintaining Quota usage statistics for all clients.

  14. case class ClientQuotaManagerConfig(quotaBytesPerSecondDefault: Long = ..., numQuotaSamples: Int = ..., quotaWindowSizeSeconds: Int = ...) extends Product with Serializable

    Permalink

    Configuration settings for quota management

    Configuration settings for quota management

    quotaBytesPerSecondDefault

    The default bytes per second quota allocated to any client

    numQuotaSamples

    The number of samples to retain in memory

    quotaWindowSizeSeconds

    The time span of each sample

  15. trait ConfigHandler extends AnyRef

    Permalink

    The ConfigHandler is used to process config change notifications received by the DynamicConfigManager

  16. class DelayedFetch extends DelayedOperation

    Permalink

    A delayed fetch operation that can be created by the replica manager and watched in the fetch operation purgatory

  17. abstract class DelayedOperation extends TimerTask with Logging

    Permalink

    An operation whose processing needs to be delayed for at most the given delayMs.

    An operation whose processing needs to be delayed for at most the given delayMs. For example a delayed produce operation could be waiting for specified number of acks; or a delayed fetch operation could be waiting for a given number of bytes to accumulate.

    The logic upon completing a delayed operation is defined in onComplete() and will be called exactly once. Once an operation is completed, isCompleted() will return true. onComplete() can be triggered by either forceComplete(), which forces calling onComplete() after delayMs if the operation is not yet completed, or tryComplete(), which first checks if the operation can be completed or not now, and if yes calls forceComplete().

    A subclass of DelayedOperation needs to provide an implementation of both onComplete() and tryComplete().

  18. trait DelayedOperationKey extends AnyRef

    Permalink

    Keys used for delayed operation metrics recording

  19. class DelayedOperationPurgatory[T <: DelayedOperation] extends Logging with KafkaMetricsGroup

    Permalink

    A helper purgatory class for bookkeeping delayed operations with a timeout, and expiring timed out operations.

  20. class DelayedProduce extends DelayedOperation

    Permalink

    A delayed produce operation that can be created by the replica manager and watched in the produce operation purgatory

  21. class DynamicConfigManager extends Logging

    Permalink

    This class initiates and carries out config changes for all entities defined in ConfigType.

    This class initiates and carries out config changes for all entities defined in ConfigType.

    It works as follows.

    Config is stored under the path: /config/entityType/entityName E.g. /config/topics/<topic_name> and /config/clients/<clientId> This znode stores the overrides for this entity (but no defaults) in properties format.

    To avoid watching all topics for changes instead we have a notification path /config/changes The DynamicConfigManager has a child watch on this path.

    To update a config we first update the config properties. Then we create a new sequential znode under the change path which contains the name of the entityType and entityName that was updated, say /config/changes/config_change_13321 The sequential znode contains data in this format: {"version" : 1, "entityType":"topic/client", "entityName" : "topic_name/client_id"} This is just a notification--the actual config change is stored only once under the /config/entityType/entityName path.

    This will fire a watcher on all brokers. This watcher works as follows. It reads all the config change notifications. It keeps track of the highest config change suffix number it has applied previously. For any previously applied change it finds it checks if this notification is larger than a static expiration time (say 10mins) and if so it deletes this notification. For any new changes it reads the new configuration, combines it with the defaults, and updates the existing config.

    Note that config is always read from the config path in zk, the notification is just a trigger to do so. So if a broker is down and misses a change that is fine--when it restarts it will be loading the full config anyway. Note also that if there are two consecutive config changes it is possible that only the last one will be applied (since by the time the broker reads the config the both changes may have been made). In this case the broker would needlessly refresh the config twice, but that is harmless.

    On restart the config manager re-processes all notifications. This will usually be wasted work, but avoids any race conditions on startup where a change might be missed between the initial config load and registering for change notifications.

  22. case class FetchDataInfo(fetchOffsetMetadata: LogOffsetMetadata, messageSet: MessageSet) extends Product with Serializable

    Permalink
  23. case class FetchMetadata(fetchMinBytes: Int, fetchOnlyLeader: Boolean, fetchOnlyCommitted: Boolean, isFromFollower: Boolean, fetchPartitionStatus: Map[TopicAndPartition, FetchPartitionStatus]) extends Product with Serializable

    Permalink

    The fetch metadata maintained by the delayed fetch operation

  24. case class FetchPartitionStatus(startOffsetMetadata: LogOffsetMetadata, fetchInfo: PartitionFetchInfo) extends Product with Serializable

    Permalink
  25. class FetcherLagMetrics extends KafkaMetricsGroup

    Permalink
  26. class FetcherLagStats extends AnyRef

    Permalink
  27. class FetcherStats extends KafkaMetricsGroup

    Permalink
  28. case class GroupKey(groupId: String) extends DelayedOperationKey with Product with Serializable

    Permalink
  29. class KafkaApis extends Logging

    Permalink

    Logic to handle the various Kafka requests

  30. case class KafkaConfig(props: Map[_, _]) extends AbstractConfig with Product with Serializable

    Permalink
  31. class KafkaHealthcheck extends Logging

    Permalink

    This class registers the broker in zookeeper to allow other brokers and consumers to detect failures.

    This class registers the broker in zookeeper to allow other brokers and consumers to detect failures. It uses an ephemeral znode with the path: /brokers/[0...N] --> advertisedHost:advertisedPort

    Right now our definition of health is fairly naive. If we register in zk we are healthy, otherwise we are dead.

  32. class KafkaRequestHandler extends Runnable with Logging

    Permalink

    A thread that answers kafka requests.

  33. class KafkaRequestHandlerPool extends Logging with KafkaMetricsGroup

    Permalink
  34. class KafkaServer extends Logging with KafkaMetricsGroup

    Permalink

    Represents the lifecycle of a single Kafka broker.

    Represents the lifecycle of a single Kafka broker. Handles all functionality required to start up and shutdown a single Kafka node.

  35. class KafkaServerStartable extends Logging

    Permalink
  36. trait LeaderElector extends Logging

    Permalink

    This trait defines a leader elector If the existing leader is dead, this class will handle automatic re-election and if it succeeds, it invokes the leader state change callback

  37. case class LogAppendResult(info: LogAppendInfo, error: Option[Throwable] = None) extends Product with Serializable

    Permalink
  38. case class LogOffsetMetadata(messageOffset: Long, segmentBaseOffset: Long = ..., relativePositionInSegment: Int = ...) extends Product with Serializable

    Permalink
  39. case class LogReadResult(info: FetchDataInfo, hw: Long, readSize: Int, isReadFromLogEnd: Boolean, error: Option[Throwable] = None) extends Product with Serializable

    Permalink
  40. case class MemberKey(groupId: String, consumerId: String) extends DelayedOperationKey with Product with Serializable

    Permalink
  41. class OffsetCheckpoint extends Logging

    Permalink

    This class saves out a map of topic/partition=>offsets to a file

  42. case class PartitionFetchState(offset: Long, delay: DelayedItem) extends Product with Serializable

    Permalink

    case class to keep partition offset and its state(active , inactive)

  43. case class ProduceMetadata(produceRequiredAcks: Short, produceStatus: Map[TopicAndPartition, ProducePartitionStatus]) extends Product with Serializable

    Permalink

    The produce metadata maintained by the delayed produce operation

  44. case class ProducePartitionStatus(requiredOffset: Long, responseStatus: ProducerResponseStatus) extends Product with Serializable

    Permalink
  45. class ReplicaFetcherManager extends AbstractFetcherManager

    Permalink
  46. class ReplicaFetcherThread extends AbstractFetcherThread

    Permalink
  47. class ReplicaManager extends Logging with KafkaMetricsGroup

    Permalink
  48. class TopicConfigHandler extends ConfigHandler

    Permalink

    The TopicConfigHandler will process topic config changes in ZK.

    The TopicConfigHandler will process topic config changes in ZK. The callback provides the topic name and the full properties set read from ZK

  49. case class TopicPartitionOperationKey(topic: String, partition: Int) extends DelayedOperationKey with Product with Serializable

    Permalink
  50. class ZookeeperLeaderElector extends LeaderElector with Logging

    Permalink

    This class handles zookeeper based leader election based on an ephemeral path.

    This class handles zookeeper based leader election based on an ephemeral path. The election module does not handle session expiration, instead it assumes the caller will handle it by probably try to re-elect again. If the existing leader is dead, this class will handle automatic re-election and if it succeeds, it invokes the leader state change callback

Value Members

  1. object AbstractFetcherThread

    Permalink
  2. object BrokerShuttingDown extends BrokerStates with Product with Serializable

    Permalink
  3. object BrokerTopicStats extends Logging

    Permalink
  4. object ClientConfigOverride

    Permalink
  5. object ClientQuotaManagerConfig extends Serializable

    Permalink
  6. object ConfigType

    Permalink

    Represents all the entities that can be configured via ZK

  7. object Defaults

    Permalink
  8. object DelayedFetchMetrics extends KafkaMetricsGroup

    Permalink
  9. object DelayedOperationKey

    Permalink
  10. object DelayedProduceMetrics extends KafkaMetricsGroup

    Permalink
  11. object KafkaConfig extends Serializable

    Permalink
  12. object KafkaServer

    Permalink
  13. object KafkaServerStartable

    Permalink
  14. object LogOffsetMetadata extends Serializable

    Permalink
  15. object LogReadResult extends Serializable

    Permalink
  16. object NotRunning extends BrokerStates with Product with Serializable

    Permalink
  17. object PendingControlledShutdown extends BrokerStates with Product with Serializable

    Permalink
  18. object RecoveringFromUncleanShutdown extends BrokerStates with Product with Serializable

    Permalink
  19. object ReplicaFetcherThread

    Permalink
  20. object ReplicaManager

    Permalink
  21. object RunningAsBroker extends BrokerStates with Product with Serializable

    Permalink
  22. object RunningAsController extends BrokerStates with Product with Serializable

    Permalink
  23. object Starting extends BrokerStates with Product with Serializable

    Permalink

Ungrouped