org.apache.hadoop.hbase.master
Class SplitLogManager

java.lang.Object
  extended by org.apache.hadoop.hbase.zookeeper.ZooKeeperListener
      extended by org.apache.hadoop.hbase.master.SplitLogManager

public class SplitLogManager
extends ZooKeeperListener

Distributes the task of log splitting to the available region servers. Coordination happens via zookeeper. For every log file that has to be split a znode is created under /hbase/splitlog. SplitLogWorkers race to grab a task. SplitLogManager monitors the task znodes that it creates using the timeoutMonitor thread. If a task's progress is slow then resubmit(String, boolean) will take away the task from the owner SplitLogWorker and the task will be upforgrabs again. When the task is done then the task's znode is deleted by SplitLogManager. Clients call splitLogDistributed(Path) to split a region server's log files. The caller thread waits in this method until all the log files have been split. All the zookeeper calls made by this class are asynchronous. This is mainly to help reduce response time seen by the callers. There is race in this design between the SplitLogManager and the SplitLogWorker. SplitLogManager might re-queue a task that has in reality already been completed by a SplitLogWorker. We rely on the idempotency of the log splitting task for correctness. It is also assumed that every log splitting task is unique and once completed (either with success or with error) it will be not be submitted again. If a task is resubmitted then there is a risk that old "delete task" can delete the re-submission.


Nested Class Summary
static interface SplitLogManager.TaskFinisher
          SplitLogManager can use objects implementing this interface to finish off a partially done task by SplitLogWorker.
 
Field Summary
 boolean ignoreZKDeleteForTesting
           
 
Fields inherited from class org.apache.hadoop.hbase.zookeeper.ZooKeeperListener
watcher
 
Constructor Summary
SplitLogManager(ZooKeeperWatcher zkw, org.apache.hadoop.conf.Configuration conf, Stoppable stopper, String serverName)
          Wrapper around SplitLogManager(ZooKeeperWatcher, Configuration, Stoppable, String, TaskFinisher) that provides a task finisher for copying recovered edits to their final destination.
SplitLogManager(ZooKeeperWatcher zkw, org.apache.hadoop.conf.Configuration conf, Stoppable stopper, String serverName, SplitLogManager.TaskFinisher tf)
          Its OK to construct this object even when region-servers are not online.
 
Method Summary
 void finishInitialization()
          Completes the initialization
 void finishInitialization(boolean masterRecovery)
           
 void nodeDataChanged(String path)
          Called when an existing node has changed data.
 long splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs)
          The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server.
 long splitLogDistributed(org.apache.hadoop.fs.Path logDir)
           
 void stop()
           
 
Methods inherited from class org.apache.hadoop.hbase.zookeeper.ZooKeeperListener
nodeChildrenChanged, nodeCreated, nodeDeleted
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

ignoreZKDeleteForTesting

public boolean ignoreZKDeleteForTesting
Constructor Detail

SplitLogManager

public SplitLogManager(ZooKeeperWatcher zkw,
                       org.apache.hadoop.conf.Configuration conf,
                       Stoppable stopper,
                       String serverName)
Wrapper around SplitLogManager(ZooKeeperWatcher, Configuration, Stoppable, String, TaskFinisher) that provides a task finisher for copying recovered edits to their final destination. The task finisher has to be robust because it can be arbitrarily restarted or called multiple times.

Parameters:
zkw -
conf -
stopper -
serverName -

SplitLogManager

public SplitLogManager(ZooKeeperWatcher zkw,
                       org.apache.hadoop.conf.Configuration conf,
                       Stoppable stopper,
                       String serverName,
                       SplitLogManager.TaskFinisher tf)
Its OK to construct this object even when region-servers are not online. It does lookup the orphan tasks in zk but it doesn't block waiting for them to be done.

Parameters:
zkw -
conf -
stopper -
serverName -
tf - task finisher
Method Detail

finishInitialization

public void finishInitialization(boolean masterRecovery)

splitLogDistributed

public long splitLogDistributed(org.apache.hadoop.fs.Path logDir)
                         throws IOException
Parameters:
logDir - one region sever hlog dir path in .logs
Returns:
cumulative size of the logfiles split
Throws:
IOException - if there was an error while splitting any log file
IOException

splitLogDistributed

public long splitLogDistributed(List<org.apache.hadoop.fs.Path> logDirs)
                         throws IOException
The caller will block until all the log files of the given region server have been processed - successfully split or an error is encountered - by an available worker region server. This method must only be called after the region servers have been brought online.

Parameters:
logDirs -
Returns:
cumulative size of the logfiles split
Throws:
IOException - if there was an error while splitting any log file

nodeDataChanged

public void nodeDataChanged(String path)
Description copied from class: ZooKeeperListener
Called when an existing node has changed data.

Overrides:
nodeDataChanged in class ZooKeeperListener
Parameters:
path - full path of the updated node

stop

public void stop()

finishInitialization

public void finishInitialization()
Completes the initialization



Copyright © 2013 The Apache Software Foundation. All Rights Reserved.