kafka.controller

KafkaController

class KafkaController extends Logging with KafkaMetricsGroup with KafkaControllerMBean

go to: companion
Inherited
  1. Hide All
  2. Show all
  1. KafkaControllerMBean
  2. KafkaMetricsGroup
  3. Logging
  4. AnyRef
  5. Any
Visibility
  1. Public
  2. All

Instance constructors

  1. new KafkaController(config: KafkaConfig, zkClient: ZkClient)

Type Members

  1. class SessionExpirationListener extends IZkStateListener with Logging

Value Members

  1. def !=(arg0: AnyRef): Boolean

    attributes: final
    definition classes: AnyRef
  2. def !=(arg0: Any): Boolean

    o != arg0 is the same as !(o == (arg0)).

    o != arg0 is the same as !(o == (arg0)).

    arg0

    the object to compare against this object for dis-equality.

    returns

    false if the receiver object is equivalent to the argument; true otherwise.

    attributes: final
    definition classes: Any
  3. def ##(): Int

    attributes: final
    definition classes: AnyRef → Any
  4. def $asInstanceOf[T0](): T0

    attributes: final
    definition classes: AnyRef
  5. def $isInstanceOf[T0](): Boolean

    attributes: final
    definition classes: AnyRef
  6. def ==(arg0: AnyRef): Boolean

    o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).

    o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    attributes: final
    definition classes: AnyRef
  7. def ==(arg0: Any): Boolean

    o == arg0 is the same as o.equals(arg0).

    o == arg0 is the same as o.equals(arg0).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    attributes: final
    definition classes: Any
  8. def asInstanceOf[T0]: T0

    This method is used to cast the receiver object to be of type T0.

    This method is used to cast the receiver object to be of type T0.

    Note that the success of a cast at runtime is modulo Scala's erasure semantics. Therefore the expression1.asInstanceOf[String] will throw a ClassCastException at runtime, while the expressionList(1).asInstanceOf[List[String]] will not. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.

    returns

    the receiver object.

    attributes: final
    definition classes: Any
  9. def clientId: String

  10. def clone(): AnyRef

    This method creates and returns a copy of the receiver object.

    This method creates and returns a copy of the receiver object.

    The default implementation of the clone method is platform dependent.

    returns

    a copy of the receiver object.

    attributes: protected
    definition classes: AnyRef
  11. val config: KafkaConfig

  12. val controllerContext: ControllerContext

  13. def debug(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  14. def debug(e: ⇒ Throwable): Any

    definition classes: Logging
  15. def debug(msg: ⇒ String): Unit

    definition classes: Logging
  16. var deleteTopicManager: TopicDeletionManager

  17. def epoch: Int

  18. def eq(arg0: AnyRef): Boolean

    This method is used to test whether the argument (arg0) is a reference to the receiver object (this).

    This method is used to test whether the argument (arg0) is a reference to the receiver object (this).

    The eq method implements an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation] on non-null instances of AnyRef: * It is reflexive: for any non-null instance x of type AnyRef, x.eq(x) returns true. * It is symmetric: for any non-null instances x and y of type AnyRef, x.eq(y) returns true if and only if y.eq(x) returns true. * It is transitive: for any non-null instances x, y, and z of type AnyRef if x.eq(y) returns true and y.eq(z) returns true, then x.eq(z) returns true.

    Additionally, the eq method has three other properties. * It is consistent: for any non-null instances x and y of type AnyRef, multiple invocations of x.eq(y) consistently returns true or consistently returns false. * For any non-null instance x of type AnyRef, x.eq(null) and null.eq(x) returns false. * null.eq(null) returns true.

    When overriding the equals or hashCode methods, it is important to ensure that their behavior is consistent with reference equality. Therefore, if two objects are references to each other (o1 eq o2), they should be equal to each other (o1 == o2) and they should hash to the same value (o1.hashCode == o2.hashCode).

    arg0

    the object to compare against this object for reference equality.

    returns

    true if the argument is a reference to the receiver object; false otherwise.

    attributes: final
    definition classes: AnyRef
  19. def equals(arg0: Any): Boolean

    This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.

    This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.

    The default implementations of this method is an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation]: * It is reflexive: for any instance x of type Any, x.equals(x) should return true. * It is symmetric: for any instances x and y of type Any, x.equals(y) should return true if and only if y.equals(x) returns true. * It is transitive: for any instances x, y, and z of type AnyRef if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) should return true.

    If you override this method, you should verify that your implementation remains an equivalence relation. Additionally, when overriding this method it is often necessary to override hashCode to ensure that objects that are "equal" (o1.equals(o2) returns true) hash to the same scala.Int (o1.hashCode.equals(o2.hashCode)).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    definition classes: AnyRef → Any
  20. def error(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  21. def error(e: ⇒ Throwable): Any

    definition classes: Logging
  22. def error(msg: ⇒ String): Unit

    definition classes: Logging
  23. def fatal(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  24. def fatal(e: ⇒ Throwable): Any

    definition classes: Logging
  25. def fatal(msg: ⇒ String): Unit

    definition classes: Logging
  26. def finalize(): Unit

    This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.

    This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.

    The details of when and if the finalize method are invoked, as well as the interaction between finalizeand non-local returns and exceptions, are all platform dependent.

    attributes: protected
    definition classes: AnyRef
  27. def getClass(): java.lang.Class[_]

    Returns a representation that corresponds to the dynamic class of the receiver object.

    Returns a representation that corresponds to the dynamic class of the receiver object.

    The nature of the representation is platform dependent.

    returns

    a representation that corresponds to the dynamic class of the receiver object.

    attributes: final
    definition classes: AnyRef
  28. def hashCode(): Int

    Returns a hash code value for the object.

    Returns a hash code value for the object.

    The default hashing algorithm is platform dependent.

    Note that it is allowed for two objects to have identical hash codes (o1.hashCode.equals(o2.hashCode)) yet not be equal (o1.equals(o2) returns false). A degenerate implementation could always return 0. However, it is required that if two objects are equal (o1.equals(o2) returns true) that they have identical hash codes (o1.hashCode.equals(o2.hashCode)). Therefore, when overriding this method, be sure to verify that the behavior is consistent with the equals method.

    returns

    the hash code value for the object.

    definition classes: AnyRef → Any
  29. def incrementControllerEpoch(zkClient: ZkClient): Unit

  30. def info(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  31. def info(e: ⇒ Throwable): Any

    definition classes: Logging
  32. def info(msg: ⇒ String): Unit

    definition classes: Logging
  33. def initiateReassignReplicasForTopicPartition(topicAndPartition: TopicAndPartition, reassignedPartitionContext: ReassignedPartitionsContext): Unit

  34. def isActive(): Boolean

    Returns true if this broker is the current controller.

    Returns true if this broker is the current controller.

  35. def isInstanceOf[T0]: Boolean

    This method is used to test whether the dynamic type of the receiver object is T0.

    This method is used to test whether the dynamic type of the receiver object is T0.

    Note that the test result of the test is modulo Scala's erasure semantics. Therefore the expression1.isInstanceOf[String] will return false, while the expression List(1).isInstanceOf[List[String]] will return true. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.

    returns

    true if the receiver object is an instance of erasure of type T0; false otherwise.

    attributes: final
    definition classes: Any
  36. var logIdent: String

    attributes: protected
    definition classes: Logging
  37. lazy val logger: Logger

    definition classes: Logging
  38. val loggerName: String

    definition classes: Logging
  39. def ne(arg0: AnyRef): Boolean

    o.ne(arg0) is the same as !(o.eq(arg0)).

    o.ne(arg0) is the same as !(o.eq(arg0)).

    arg0

    the object to compare against this object for reference dis-equality.

    returns

    false if the argument is not a reference to the receiver object; true otherwise.

    attributes: final
    definition classes: AnyRef
  40. def newGauge[T](name: String, metric: Gauge[T]): Gauge[T]

    definition classes: KafkaMetricsGroup
  41. def newHistogram(name: String, biased: Boolean = true): Histogram

    definition classes: KafkaMetricsGroup
  42. def newMeter(name: String, eventType: String, timeUnit: TimeUnit): Meter

    definition classes: KafkaMetricsGroup
  43. def newTimer(name: String, durationUnit: TimeUnit, rateUnit: TimeUnit): Timer

    definition classes: KafkaMetricsGroup
  44. def notify(): Unit

    Wakes up a single thread that is waiting on the receiver object's monitor.

    Wakes up a single thread that is waiting on the receiver object's monitor.

    attributes: final
    definition classes: AnyRef
  45. def notifyAll(): Unit

    Wakes up all threads that are waiting on the receiver object's monitor.

    Wakes up all threads that are waiting on the receiver object's monitor.

    attributes: final
    definition classes: AnyRef
  46. val offlinePartitionSelector: OfflinePartitionLeaderSelector

  47. def onBrokerFailure(deadBrokers: Seq[Int]): Unit

    This callback is invoked by the replica state machine's broker change listener with the list of failed brokers as input.

    This callback is invoked by the replica state machine's broker change listener with the list of failed brokers as input. It does the following - 1. Mark partitions with dead leaders as offline 2. Triggers the OnlinePartition state change for all new/offline partitions 3. Invokes the OfflineReplica state change on the input list of newly started brokers

    Note that we don't need to refresh the leader/isr cache for all topic/partitions at this point. This is because the partition state machine will refresh our cache for us when performing leader election for all new/offline partitions coming online.

  48. def onBrokerStartup(newBrokers: Seq[Int]): Unit

    This callback is invoked by the replica state machine's broker change listener, with the list of newly started brokers as input.

    This callback is invoked by the replica state machine's broker change listener, with the list of newly started brokers as input. It does the following - 1. Triggers the OnlinePartition state change for all new/offline partitions 2. It checks whether there are reassigned replicas assigned to any newly started brokers. If so, it performs the reassignment logic for each topic/partition.

    Note that we don't need to refresh the leader/isr cache for all topic/partitions at this point for two reasons: 1. The partition state machine, when triggering online state change, will refresh leader and ISR for only those partitions currently new or offline (rather than every partition this controller is aware of) 2. Even if we do refresh the cache, there is no guarantee that by the time the leader and ISR request reaches every broker that it is still valid. Brokers check the leader epoch to determine validity of the request.

  49. def onControllerFailover(): Unit

    This callback is invoked by the zookeeper leader elector on electing the current broker as the new controller.

    This callback is invoked by the zookeeper leader elector on electing the current broker as the new controller. It does the following things on the become-controller state change - 1. Register controller epoch changed listener 2. Increments the controller epoch 3. Initializes the controller's context object that holds cache objects for current topics, live brokers and leaders for all existing partitions. 4. Starts the controller's channel manager 5. Starts the replica state machine 6. Starts the partition state machine If it encounters any unexpected exception/error while becoming controller, it resigns as the current controller. This ensures another controller election will be triggered and there will always be an actively serving controller

  50. def onControllerResignation(): Unit

    This callback is invoked by the zookeeper leader elector when the current broker resigns as the controller.

    This callback is invoked by the zookeeper leader elector when the current broker resigns as the controller. This is required to clean up internal controller data structures

  51. def onNewPartitionCreation(newPartitions: Set[TopicAndPartition]): Unit

    This callback is invoked by the topic change callback with the list of failed brokers as input.

    This callback is invoked by the topic change callback with the list of failed brokers as input. It does the following - 1. Move the newly created partitions to the NewPartition state 2. Move the newly created partitions from NewPartition->OnlinePartition state

  52. def onNewTopicCreation(topics: Set[String], newPartitions: Set[TopicAndPartition]): Unit

    This callback is invoked by the partition state machine's topic change listener with the list of new topics and partitions as input.

    This callback is invoked by the partition state machine's topic change listener with the list of new topics and partitions as input. It does the following - 1. Registers partition change listener. This is not required until KAFKA-347 2. Invokes the new partition callback 3. Send metadata request with the new topic to all brokers so they allow requests for that topic to be served

  53. def onPartitionReassignment(topicAndPartition: TopicAndPartition, reassignedPartitionContext: ReassignedPartitionsContext): Unit

    This callback is invoked by the reassigned partitions listener.

    This callback is invoked by the reassigned partitions listener. When an admin command initiates a partition reassignment, it creates the /admin/reassign_partitions path that triggers the zookeeper listener. Reassigning replicas for a partition goes through a few steps listed in the code. RAR = Reassigned replicas OAR = Original list of replicas for partition AR = current assigned replicas

    1. Update AR in ZK with OAR + RAR. 2. Send LeaderAndIsr request to every replica in OAR + RAR (with AR as OAR + RAR). We do this by forcing an update of the leader epoch in zookeeper. 3. Start new replicas RAR - OAR by moving replicas in RAR - OAR to NewReplica state. 4. Wait until all replicas in RAR are in sync with the leader. 5 Move all replicas in RAR to OnlineReplica state. 6. Set AR to RAR in memory. 7. If the leader is not in RAR, elect a new leader from RAR. If new leader needs to be elected from RAR, a LeaderAndIsr will be sent. If not, then leader epoch will be incremented in zookeeper and a LeaderAndIsr request will be sent. In any case, the LeaderAndIsr request will have AR = RAR. This will prevent the leader from adding any replica in RAR - OAR back in the isr. 8. Move all replicas in OAR - RAR to OfflineReplica state. As part of OfflineReplica state change, we shrink the isr to remove OAR - RAR in zookeeper and sent a LeaderAndIsr ONLY to the Leader to notify it of the shrunk isr. After that, we send a StopReplica (delete = false) to the replicas in OAR - RAR. 9. Move all replicas in OAR - RAR to NonExistentReplica state. This will send a StopReplica (delete = false) to the replicas in OAR - RAR to physically delete the replicas on disk. 10. Update AR in ZK with RAR. 11. Update the /admin/reassign_partitions path in ZK to remove this partition. 12. After electing leader, the replicas and isr information changes. So resend the update metadata request to every broker.

    For example, if OAR = {1, 2, 3} and RAR = {4,5,6}, the values in the assigned replica (AR) and leader/isr path in ZK may go through the following transition. AR leader/isr {1,2,3} 1/{1,2,3} (initial state) {1,2,3,4,5,6} 1/{1,2,3} (step 2) {1,2,3,4,5,6} 1/{1,2,3,4,5,6} (step 4) {1,2,3,4,5,6} 4/{1,2,3,4,5,6} (step 7) {1,2,3,4,5,6} 4/{4,5,6} (step 8) {4,5,6} 4/{4,5,6} (step 10)

    Note that we have to update AR in ZK with RAR last since it's the only place where we store OAR persistently. This way, if the controller crashes before that step, we can still recover.

  54. def onPreferredReplicaElection(partitions: Set[TopicAndPartition], isTriggeredByAutoRebalance: Boolean = false): Unit

  55. val partitionStateMachine: PartitionStateMachine

  56. def removePartitionFromReassignedPartitions(topicAndPartition: TopicAndPartition): Unit

  57. def removePartitionsFromPreferredReplicaElection(partitionsToBeRemoved: Set[TopicAndPartition], isTriggeredByAutoRebalance: Boolean): Unit

  58. def removeReplicaFromIsr(topic: String, partition: Int, replicaId: Int): Option[LeaderIsrAndControllerEpoch]

    Removes a given partition replica from the ISR; if it is not the current leader and there are sufficient remaining replicas in ISR.

    Removes a given partition replica from the ISR; if it is not the current leader and there are sufficient remaining replicas in ISR.

    topic

    topic

    partition

    partition

    replicaId

    replica Id

    returns

    the new leaderAndIsr (with the replica removed if it was present), or None if leaderAndIsr is empty.

  59. val replicaStateMachine: ReplicaStateMachine

  60. def sendRequest(brokerId: Int, request: RequestOrResponse, callback: (RequestOrResponse) ⇒ Unit = null): Unit

  61. def sendUpdateMetadataRequest(brokers: Seq[Int], partitions: Set[TopicAndPartition] = scala.collection.Set.empty[kafka.common.TopicAndPartition]): Unit

    Send the leader information for selected partitions to selected brokers so that they can correctly respond to metadata requests

    Send the leader information for selected partitions to selected brokers so that they can correctly respond to metadata requests

    brokers

    The brokers that the update metadata request should be sent to

  62. def shutdown(): Unit

    Invoked when the controller module of a Kafka server is shutting down.

    Invoked when the controller module of a Kafka server is shutting down. If the broker was the current controller, it shuts down the partition and replica state machines. If not, those are a no-op. In addition to that, it also shuts down the controller channel manager, if one exists (i.e. if it was the current controller)

  63. def shutdownBroker(id: Int): Set[TopicAndPartition]

    On clean shutdown, the controller first determines the partitions that the shutting down broker leads, and moves leadership of those partitions to another broker that is in that partition's ISR.

    On clean shutdown, the controller first determines the partitions that the shutting down broker leads, and moves leadership of those partitions to another broker that is in that partition's ISR.

    id

    Id of the broker to shutdown.

    returns

    The number of partitions that the broker still leads.

    definition classes: KafkaControllerKafkaControllerMBean
  64. def startup(): Unit

    Invoked when the controller module of a Kafka server is started up.

    Invoked when the controller module of a Kafka server is started up. This does not assume that the current broker is the controller. It merely registers the session expiration listener and starts the controller leader elector

  65. def swallow(action: ⇒ Unit): Unit

    definition classes: Logging
  66. def swallowDebug(action: ⇒ Unit): Unit

    definition classes: Logging
  67. def swallowError(action: ⇒ Unit): Unit

    definition classes: Logging
  68. def swallowInfo(action: ⇒ Unit): Unit

    definition classes: Logging
  69. def swallowTrace(action: ⇒ Unit): Unit

    definition classes: Logging
  70. def swallowWarn(action: ⇒ Unit): Unit

    definition classes: Logging
  71. def synchronized[T0](arg0: T0): T0

    attributes: final
    definition classes: AnyRef
  72. def toString(): String

    Returns a string representation of the object.

    Returns a string representation of the object.

    The default representation is platform dependent.

    returns

    a string representation of the object.

    definition classes: AnyRef → Any
  73. def trace(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  74. def trace(e: ⇒ Throwable): Any

    definition classes: Logging
  75. def trace(msg: ⇒ String): Unit

    definition classes: Logging
  76. def updateAssignedReplicasForPartition(topicAndPartition: TopicAndPartition, newReplicaAssignmentForTopic: Map[TopicAndPartition, Seq[Int]]): Unit

  77. def wait(): Unit

    attributes: final
    definition classes: AnyRef
  78. def wait(arg0: Long, arg1: Int): Unit

    attributes: final
    definition classes: AnyRef
  79. def wait(arg0: Long): Unit

    attributes: final
    definition classes: AnyRef
  80. def warn(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  81. def warn(e: ⇒ Throwable): Any

    definition classes: Logging
  82. def warn(msg: ⇒ String): Unit

    definition classes: Logging