The topics that are queued up for deletion in zookeeper at the time of controller failover
The topics ineligible for deletion due to any of the conditions mentioned in #3 above
Invoked by the child change listener on /admin/delete_topics to queue up the topics for deletion.
Invoked by the child change listener on /admin/delete_topics to queue up the topics for deletion. The topic gets added to the topicsToBeDeleted list and only gets removed from the list when the topic deletion has completed successfully i.e. all replicas of all partitions of that topic are deleted successfully.
Topics that should be deleted
Invoked when a broker that hosts replicas for topics to be deleted goes down.
Invoked when a broker that hosts replicas for topics to be deleted goes down. Also invoked when the callback for StopReplicaResponse receives an error code for the replicas of a topic to be deleted. As part of this, the replicas are moved from ReplicaDeletionStarted to ReplicaDeletionIneligible state. Also, the topic is added to the list of topics ineligible for deletion until further notice. The delete topic thread is notified so it can retry topic deletion if it has received a response for all replicas of a topic to be deleted
Replicas for which deletion has failed
Halt delete topic if - 1.
Halt delete topic if - 1. replicas being down 2. partition reassignment in progress for some partitions of the topic 3. preferred replica election in progress for some partitions of the topic
Topics that should be marked ineligible for deletion. No op if the topic is was not previously queued up for deletion
Invoked when any event that can possibly resume topic deletion occurs.
Invoked when any event that can possibly resume topic deletion occurs. These events include - 1. New broker starts up. Any replicas belonging to topics queued up for deletion can be deleted since the broker is up 2. Partition reassignment completes. Any partitions belonging to topics queued up for deletion finished reassignment 3. Preferred replica election completes. Any partitions belonging to topics queued up for deletion finished preferred replica election
Topics for which deletion can be resumed
Invoked when the current controller resigns.
Invoked when the current controller resigns. At this time, all state for topic deletion should be cleared.
Invoked at the end of new controller initiation
This manages the state machine for topic deletion. 1. TopicCommand issues topic deletion by creating a new admin path /admin/delete_topics/<topic> 2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics 3. The controller has a background thread that handles topic deletion. The purpose of having this background thread is to accommodate the TTL feature, when we have it. This thread is signaled whenever deletion for a topic needs to be started or resumed. Currently, a topic's deletion can be started only by the onPartitionDeletion callback on the controller. In the future, it can be triggered based on the configured TTL for the topic. A topic will be ineligible for deletion in the following scenarios - 3.1 broker hosting one of the replicas for that topic goes down 3.2 partition reassignment for partitions of that topic is in progress 3.3 preferred replica election for partitions of that topic is in progress (though this is not strictly required since it holds the controller lock for the entire duration from start to end) 4. Topic deletion is resumed when - 4.1 broker hosting one of the replicas for that topic is started 4.2 preferred replica election for partitions of that topic completes 4.3 partition reassignment for partitions of that topic completes 5. Every replica for a topic being deleted is in either of the 3 states - 5.1 TopicDeletionStarted (Replica enters TopicDeletionStarted phase when the onPartitionDeletion callback is invoked. This happens when the child change watch for /admin/delete_topics fires on the controller. As part of this state change, the controller sends StopReplicaRequests to all replicas. It registers a callback for the StopReplicaResponse when deletePartition=true thereby invoking a callback when a response for delete replica is received from every replica) 5.2 TopicDeletionSuccessful (deleteTopicStopReplicaCallback() moves replicas from TopicDeletionStarted->TopicDeletionSuccessful depending on the error codes in StopReplicaResponse) 5.3 TopicDeletionFailed. (deleteTopicStopReplicaCallback() moves replicas from TopicDeletionStarted->TopicDeletionFailed depending on the error codes in StopReplicaResponse. In general, if a broker dies and if it hosted replicas for topics being deleted, the controller marks the respective replicas in TopicDeletionFailed state in the onBrokerFailure callback. The reason is that if a broker fails before the request is sent and after the replica is in TopicDeletionStarted state, it is possible that the replica will mistakenly remain in TopicDeletionStarted state and topic deletion will not be retried when the broker comes back up.) 6. The delete topic thread marks a topic successfully deleted only if all replicas are in TopicDeletionSuccessful state and it starts the topic deletion teardown mode where it deletes all topic state from the controllerContext as well as from zookeeper. This is the only time the /brokers/topics/<topic> path gets deleted. On the other hand, if no replica is in TopicDeletionStarted state and at least one replica is in TopicDeletionFailed state, then it marks the topic for deletion retry.