kafka.controller

TopicDeletionManager

class TopicDeletionManager extends Logging

This manages the state machine for topic deletion. 1. TopicCommand issues topic deletion by creating a new admin path /admin/delete_topics/<topic> 2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics 3. The controller has a background thread that handles topic deletion. The purpose of having this background thread is to accommodate the TTL feature, when we have it. This thread is signaled whenever deletion for a topic needs to be started or resumed. Currently, a topic's deletion can be started only by the onPartitionDeletion callback on the controller. In the future, it can be triggered based on the configured TTL for the topic. A topic will be ineligible for deletion in the following scenarios - 3.1 broker hosting one of the replicas for that topic goes down 3.2 partition reassignment for partitions of that topic is in progress 3.3 preferred replica election for partitions of that topic is in progress (though this is not strictly required since it holds the controller lock for the entire duration from start to end) 4. Topic deletion is resumed when - 4.1 broker hosting one of the replicas for that topic is started 4.2 preferred replica election for partitions of that topic completes 4.3 partition reassignment for partitions of that topic completes 5. Every replica for a topic being deleted is in either of the 3 states - 5.1 TopicDeletionStarted (Replica enters TopicDeletionStarted phase when the onPartitionDeletion callback is invoked. This happens when the child change watch for /admin/delete_topics fires on the controller. As part of this state change, the controller sends StopReplicaRequests to all replicas. It registers a callback for the StopReplicaResponse when deletePartition=true thereby invoking a callback when a response for delete replica is received from every replica) 5.2 TopicDeletionSuccessful (deleteTopicStopReplicaCallback() moves replicas from TopicDeletionStarted->TopicDeletionSuccessful depending on the error codes in StopReplicaResponse) 5.3 TopicDeletionFailed. (deleteTopicStopReplicaCallback() moves replicas from TopicDeletionStarted->TopicDeletionFailed depending on the error codes in StopReplicaResponse. In general, if a broker dies and if it hosted replicas for topics being deleted, the controller marks the respective replicas in TopicDeletionFailed state in the onBrokerFailure callback. The reason is that if a broker fails before the request is sent and after the replica is in TopicDeletionStarted state, it is possible that the replica will mistakenly remain in TopicDeletionStarted state and topic deletion will not be retried when the broker comes back up.) 6. The delete topic thread marks a topic successfully deleted only if all replicas are in TopicDeletionSuccessful state and it starts the topic deletion teardown mode where it deletes all topic state from the controllerContext as well as from zookeeper. This is the only time the /brokers/topics/<topic> path gets deleted. On the other hand, if no replica is in TopicDeletionStarted state and at least one replica is in TopicDeletionFailed state, then it marks the topic for deletion retry.

go to: companion
Inherited
  1. Hide All
  2. Show all
  1. Logging
  2. AnyRef
  3. Any
Visibility
  1. Public
  2. All

Instance constructors

  1. new TopicDeletionManager(controller: KafkaController, initialTopicsToBeDeleted: Set[String] = scala.collection.Set.empty[String], initialTopicsIneligibleForDeletion: Set[String] = scala.collection.Set.empty[String])

Type Members

  1. class DeleteTopicsThread extends ShutdownableThread

Value Members

  1. def !=(arg0: AnyRef): Boolean

    attributes: final
    definition classes: AnyRef
  2. def !=(arg0: Any): Boolean

    o != arg0 is the same as !(o == (arg0)).

    o != arg0 is the same as !(o == (arg0)).

    arg0

    the object to compare against this object for dis-equality.

    returns

    false if the receiver object is equivalent to the argument; true otherwise.

    attributes: final
    definition classes: Any
  3. def ##(): Int

    attributes: final
    definition classes: AnyRef → Any
  4. def $asInstanceOf[T0](): T0

    attributes: final
    definition classes: AnyRef
  5. def $isInstanceOf[T0](): Boolean

    attributes: final
    definition classes: AnyRef
  6. def ==(arg0: AnyRef): Boolean

    o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).

    o == arg0 is the same as if (o eq null) arg0 eq null else o.equals(arg0).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    attributes: final
    definition classes: AnyRef
  7. def ==(arg0: Any): Boolean

    o == arg0 is the same as o.equals(arg0).

    o == arg0 is the same as o.equals(arg0).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    attributes: final
    definition classes: Any
  8. def asInstanceOf[T0]: T0

    This method is used to cast the receiver object to be of type T0.

    This method is used to cast the receiver object to be of type T0.

    Note that the success of a cast at runtime is modulo Scala's erasure semantics. Therefore the expression1.asInstanceOf[String] will throw a ClassCastException at runtime, while the expressionList(1).asInstanceOf[List[String]] will not. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.

    returns

    the receiver object.

    attributes: final
    definition classes: Any
  9. def clone(): AnyRef

    This method creates and returns a copy of the receiver object.

    This method creates and returns a copy of the receiver object.

    The default implementation of the clone method is platform dependent.

    returns

    a copy of the receiver object.

    attributes: protected
    definition classes: AnyRef
  10. val controllerContext: ControllerContext

  11. def debug(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  12. def debug(e: ⇒ Throwable): Any

    definition classes: Logging
  13. def debug(msg: ⇒ String): Unit

    definition classes: Logging
  14. val deleteLock: ReentrantLock

  15. val deleteTopicStateChanged: AtomicBoolean

  16. val deleteTopicsCond: Condition

  17. var deleteTopicsThread: DeleteTopicsThread

  18. def enqueueTopicsForDeletion(topics: Set[String]): Unit

    Invoked by the child change listener on /admin/delete_topics to queue up the topics for deletion.

    Invoked by the child change listener on /admin/delete_topics to queue up the topics for deletion. The topic gets added to the topicsToBeDeleted list and only gets removed from the list when the topic deletion has completed successfully i.e. all replicas of all partitions of that topic are deleted successfully.

    topics

    Topics that should be deleted

  19. def eq(arg0: AnyRef): Boolean

    This method is used to test whether the argument (arg0) is a reference to the receiver object (this).

    This method is used to test whether the argument (arg0) is a reference to the receiver object (this).

    The eq method implements an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation] on non-null instances of AnyRef: * It is reflexive: for any non-null instance x of type AnyRef, x.eq(x) returns true. * It is symmetric: for any non-null instances x and y of type AnyRef, x.eq(y) returns true if and only if y.eq(x) returns true. * It is transitive: for any non-null instances x, y, and z of type AnyRef if x.eq(y) returns true and y.eq(z) returns true, then x.eq(z) returns true.

    Additionally, the eq method has three other properties. * It is consistent: for any non-null instances x and y of type AnyRef, multiple invocations of x.eq(y) consistently returns true or consistently returns false. * For any non-null instance x of type AnyRef, x.eq(null) and null.eq(x) returns false. * null.eq(null) returns true.

    When overriding the equals or hashCode methods, it is important to ensure that their behavior is consistent with reference equality. Therefore, if two objects are references to each other (o1 eq o2), they should be equal to each other (o1 == o2) and they should hash to the same value (o1.hashCode == o2.hashCode).

    arg0

    the object to compare against this object for reference equality.

    returns

    true if the argument is a reference to the receiver object; false otherwise.

    attributes: final
    definition classes: AnyRef
  20. def equals(arg0: Any): Boolean

    This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.

    This method is used to compare the receiver object (this) with the argument object (arg0) for equivalence.

    The default implementations of this method is an [http://en.wikipedia.org/wiki/Equivalence_relation equivalence relation]: * It is reflexive: for any instance x of type Any, x.equals(x) should return true. * It is symmetric: for any instances x and y of type Any, x.equals(y) should return true if and only if y.equals(x) returns true. * It is transitive: for any instances x, y, and z of type AnyRef if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) should return true.

    If you override this method, you should verify that your implementation remains an equivalence relation. Additionally, when overriding this method it is often necessary to override hashCode to ensure that objects that are "equal" (o1.equals(o2) returns true) hash to the same scala.Int (o1.hashCode.equals(o2.hashCode)).

    arg0

    the object to compare against this object for equality.

    returns

    true if the receiver object is equivalent to the argument; false otherwise.

    definition classes: AnyRef → Any
  21. def error(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  22. def error(e: ⇒ Throwable): Any

    definition classes: Logging
  23. def error(msg: ⇒ String): Unit

    definition classes: Logging
  24. def failReplicaDeletion(replicas: Set[PartitionAndReplica]): Unit

    Invoked when a broker that hosts replicas for topics to be deleted goes down.

    Invoked when a broker that hosts replicas for topics to be deleted goes down. Also invoked when the callback for StopReplicaResponse receives an error code for the replicas of a topic to be deleted. As part of this, the replicas are moved from ReplicaDeletionStarted to ReplicaDeletionIneligible state. Also, the topic is added to the list of topics ineligible for deletion until further notice. The delete topic thread is notified so it can retry topic deletion if it has received a response for all replicas of a topic to be deleted

    replicas

    Replicas for which deletion has failed

  25. def fatal(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  26. def fatal(e: ⇒ Throwable): Any

    definition classes: Logging
  27. def fatal(msg: ⇒ String): Unit

    definition classes: Logging
  28. def finalize(): Unit

    This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.

    This method is called by the garbage collector on the receiver object when garbage collection determines that there are no more references to the object.

    The details of when and if the finalize method are invoked, as well as the interaction between finalizeand non-local returns and exceptions, are all platform dependent.

    attributes: protected
    definition classes: AnyRef
  29. def getClass(): java.lang.Class[_]

    Returns a representation that corresponds to the dynamic class of the receiver object.

    Returns a representation that corresponds to the dynamic class of the receiver object.

    The nature of the representation is platform dependent.

    returns

    a representation that corresponds to the dynamic class of the receiver object.

    attributes: final
    definition classes: AnyRef
  30. def hashCode(): Int

    Returns a hash code value for the object.

    Returns a hash code value for the object.

    The default hashing algorithm is platform dependent.

    Note that it is allowed for two objects to have identical hash codes (o1.hashCode.equals(o2.hashCode)) yet not be equal (o1.equals(o2) returns false). A degenerate implementation could always return 0. However, it is required that if two objects are equal (o1.equals(o2) returns true) that they have identical hash codes (o1.hashCode.equals(o2.hashCode)). Therefore, when overriding this method, be sure to verify that the behavior is consistent with the equals method.

    returns

    the hash code value for the object.

    definition classes: AnyRef → Any
  31. def info(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  32. def info(e: ⇒ Throwable): Any

    definition classes: Logging
  33. def info(msg: ⇒ String): Unit

    definition classes: Logging
  34. val isDeleteTopicEnabled: Boolean

  35. def isInstanceOf[T0]: Boolean

    This method is used to test whether the dynamic type of the receiver object is T0.

    This method is used to test whether the dynamic type of the receiver object is T0.

    Note that the test result of the test is modulo Scala's erasure semantics. Therefore the expression1.isInstanceOf[String] will return false, while the expression List(1).isInstanceOf[List[String]] will return true. In the latter example, because the type argument is erased as part of compilation it is not possible to check whether the contents of the list are of the requested typed.

    returns

    true if the receiver object is an instance of erasure of type T0; false otherwise.

    attributes: final
    definition classes: Any
  36. def isTopicDeletionInProgress(topic: String): Boolean

  37. def isTopicIneligibleForDeletion(topic: String): Boolean

  38. def isTopicQueuedUpForDeletion(topic: String): Boolean

  39. var logIdent: String

    attributes: protected
    definition classes: Logging
  40. lazy val logger: Logger

    definition classes: Logging
  41. val loggerName: String

    definition classes: Logging
  42. def markTopicIneligibleForDeletion(topics: Set[String]): Unit

    Halt delete topic if - 1.

    Halt delete topic if - 1. replicas being down 2. partition reassignment in progress for some partitions of the topic 3. preferred replica election in progress for some partitions of the topic

    topics

    Topics that should be marked ineligible for deletion. No op if the topic is was not previously queued up for deletion

  43. def ne(arg0: AnyRef): Boolean

    o.ne(arg0) is the same as !(o.eq(arg0)).

    o.ne(arg0) is the same as !(o.eq(arg0)).

    arg0

    the object to compare against this object for reference dis-equality.

    returns

    false if the argument is not a reference to the receiver object; true otherwise.

    attributes: final
    definition classes: AnyRef
  44. def notify(): Unit

    Wakes up a single thread that is waiting on the receiver object's monitor.

    Wakes up a single thread that is waiting on the receiver object's monitor.

    attributes: final
    definition classes: AnyRef
  45. def notifyAll(): Unit

    Wakes up all threads that are waiting on the receiver object's monitor.

    Wakes up all threads that are waiting on the receiver object's monitor.

    attributes: final
    definition classes: AnyRef
  46. val partitionStateMachine: PartitionStateMachine

  47. val partitionsToBeDeleted: Set[TopicAndPartition]

  48. val replicaStateMachine: ReplicaStateMachine

  49. def resumeDeletionForTopics(topics: Set[String] = scala.collection.Set.empty[String]): Unit

    Invoked when any event that can possibly resume topic deletion occurs.

    Invoked when any event that can possibly resume topic deletion occurs. These events include - 1. New broker starts up. Any replicas belonging to topics queued up for deletion can be deleted since the broker is up 2. Partition reassignment completes. Any partitions belonging to topics queued up for deletion finished reassignment 3. Preferred replica election completes. Any partitions belonging to topics queued up for deletion finished preferred replica election

    topics

    Topics for which deletion can be resumed

  50. def shutdown(): Unit

    Invoked when the current controller resigns.

    Invoked when the current controller resigns. At this time, all state for topic deletion should be cleared

  51. def start(): Unit

    Invoked at the end of new controller initiation

    Invoked at the end of new controller initiation

  52. def swallow(action: ⇒ Unit): Unit

    definition classes: Logging
  53. def swallowDebug(action: ⇒ Unit): Unit

    definition classes: Logging
  54. def swallowError(action: ⇒ Unit): Unit

    definition classes: Logging
  55. def swallowInfo(action: ⇒ Unit): Unit

    definition classes: Logging
  56. def swallowTrace(action: ⇒ Unit): Unit

    definition classes: Logging
  57. def swallowWarn(action: ⇒ Unit): Unit

    definition classes: Logging
  58. def synchronized[T0](arg0: T0): T0

    attributes: final
    definition classes: AnyRef
  59. def toString(): String

    Returns a string representation of the object.

    Returns a string representation of the object.

    The default representation is platform dependent.

    returns

    a string representation of the object.

    definition classes: AnyRef → Any
  60. val topicsIneligibleForDeletion: Set[String]

  61. val topicsToBeDeleted: Set[String]

  62. def trace(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  63. def trace(e: ⇒ Throwable): Any

    definition classes: Logging
  64. def trace(msg: ⇒ String): Unit

    definition classes: Logging
  65. def wait(): Unit

    attributes: final
    definition classes: AnyRef
  66. def wait(arg0: Long, arg1: Int): Unit

    attributes: final
    definition classes: AnyRef
  67. def wait(arg0: Long): Unit

    attributes: final
    definition classes: AnyRef
  68. def warn(msg: ⇒ String, e: ⇒ Throwable): Unit

    definition classes: Logging
  69. def warn(e: ⇒ Throwable): Any

    definition classes: Logging
  70. def warn(msg: ⇒ String): Unit

    definition classes: Logging