org.apache.hadoop.hbase.replication.regionserver
Class ReplicationSource

java.lang.Object
  extended by java.lang.Thread
      extended by org.apache.hadoop.hbase.replication.regionserver.ReplicationSource
All Implemented Interfaces:
Runnable, ReplicationSourceInterface

public class ReplicationSource
extends Thread
implements ReplicationSourceInterface

Class that handles the source of a replication stream. Currently does not handle more than 1 slave For each slave cluster it selects a random number of peers using a replication ratio. For example, if replication ration = 0.1 and slave cluster has 100 region servers, 10 will be selected.

A stream is considered down when we cannot contact a region server on the peer cluster for more than 55 seconds by default.


Nested Class Summary
static class ReplicationSource.LogsComparator
          Comparator used to compare logs together based on their start time
 
Nested classes/interfaces inherited from class java.lang.Thread
Thread.State, Thread.UncaughtExceptionHandler
 
Field Summary
 
Fields inherited from class java.lang.Thread
MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY
 
Constructor Summary
ReplicationSource()
           
 
Method Summary
 void enqueueLog(org.apache.hadoop.fs.Path log)
          Add a log to the list of logs to replicate
 org.apache.hadoop.fs.Path getCurrentPath()
          Get the current log that's replicated
protected  boolean getNextPath()
          Poll for the next path
 String getPeerClusterId()
          Get the id that the source is replicating to.
 String getPeerClusterZnode()
          Get the id that the source is replicating to
 void init(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ReplicationSourceManager manager, Stoppable stopper, AtomicBoolean replicating, String peerClusterZnode)
          Instantiation method used by region servers
 boolean isSlaveDown()
          Check if the slave is down by trying to establish a connection
protected  boolean openReader(int sleepMultiplier)
          Open a reader on the current path
protected  boolean processEndOfFile()
          If the queue isn't empty, switch to the next one Else if this is a recovered queue, it means we're done! Else we'll just continue to try reading the log file
protected  boolean readAllEntriesToReplicateOrNextFile()
          Read all the entries from the current log files and retain those that need to be replicated.
protected  void removeNonReplicableEdits(WALEdit edit)
          We only want KVs that are scoped other than local
 void run()
           
 void setSourceEnabled(boolean status)
          Set if this source is enabled or disabled
protected  void shipEdits()
          Do the shipping logic
protected  boolean sleepForRetries(String msg, int sleepMultiplier)
          Do the sleeping logic
 void startup()
          Start the replication
 void terminate(String reason)
          End the replication
 void terminate(String reason, Exception cause)
          End the replication
 
Methods inherited from class java.lang.Thread
activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield
 
Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Constructor Detail

ReplicationSource

public ReplicationSource()
Method Detail

init

public void init(org.apache.hadoop.conf.Configuration conf,
                 org.apache.hadoop.fs.FileSystem fs,
                 ReplicationSourceManager manager,
                 Stoppable stopper,
                 AtomicBoolean replicating,
                 String peerClusterZnode)
          throws IOException
Instantiation method used by region servers

Specified by:
init in interface ReplicationSourceInterface
Parameters:
conf - configuration to use
fs - file system to use
manager - replication manager to ping to
stopper - the atomic boolean to use to stop the regionserver
replicating - the atomic boolean that starts/stops replication
peerClusterZnode - the name of our znode
Throws:
IOException

enqueueLog

public void enqueueLog(org.apache.hadoop.fs.Path log)
Description copied from interface: ReplicationSourceInterface
Add a log to the list of logs to replicate

Specified by:
enqueueLog in interface ReplicationSourceInterface
Parameters:
log - path to the log to replicate

run

public void run()
Specified by:
run in interface Runnable
Overrides:
run in class Thread

readAllEntriesToReplicateOrNextFile

protected boolean readAllEntriesToReplicateOrNextFile()
                                               throws IOException
Read all the entries from the current log files and retain those that need to be replicated. Else, process the end of the current file.

Returns:
true if we got nothing and went to the next file, false if we got entries
Throws:
IOException

getNextPath

protected boolean getNextPath()
Poll for the next path

Returns:
true if a path was obtained, false if not

openReader

protected boolean openReader(int sleepMultiplier)
Open a reader on the current path

Parameters:
sleepMultiplier - by how many times the default sleeping time is augmented
Returns:
true if we should continue with that file, false if we are over with it

sleepForRetries

protected boolean sleepForRetries(String msg,
                                  int sleepMultiplier)
Do the sleeping logic

Parameters:
msg - Why we sleep
sleepMultiplier - by how many times the default sleeping time is augmented
Returns:
True if sleepMultiplier is < maxRetriesMultiplier

removeNonReplicableEdits

protected void removeNonReplicableEdits(WALEdit edit)
We only want KVs that are scoped other than local

Parameters:
edit - The KV to check for replication

shipEdits

protected void shipEdits()
Do the shipping logic


processEndOfFile

protected boolean processEndOfFile()
If the queue isn't empty, switch to the next one Else if this is a recovered queue, it means we're done! Else we'll just continue to try reading the log file

Returns:
true if we're done with the current file, false if we should continue trying to read from it

startup

public void startup()
Description copied from interface: ReplicationSourceInterface
Start the replication

Specified by:
startup in interface ReplicationSourceInterface

terminate

public void terminate(String reason)
Description copied from interface: ReplicationSourceInterface
End the replication

Specified by:
terminate in interface ReplicationSourceInterface
Parameters:
reason - why it's terminating

terminate

public void terminate(String reason,
                      Exception cause)
Description copied from interface: ReplicationSourceInterface
End the replication

Specified by:
terminate in interface ReplicationSourceInterface
Parameters:
reason - why it's terminating
cause - the error that's causing it

isSlaveDown

public boolean isSlaveDown()
                    throws InterruptedException
Check if the slave is down by trying to establish a connection

Returns:
true if down, false if up
Throws:
InterruptedException

getPeerClusterZnode

public String getPeerClusterZnode()
Description copied from interface: ReplicationSourceInterface
Get the id that the source is replicating to

Specified by:
getPeerClusterZnode in interface ReplicationSourceInterface
Returns:
peer cluster id

getPeerClusterId

public String getPeerClusterId()
Description copied from interface: ReplicationSourceInterface
Get the id that the source is replicating to.

Specified by:
getPeerClusterId in interface ReplicationSourceInterface
Returns:
peer cluster id

getCurrentPath

public org.apache.hadoop.fs.Path getCurrentPath()
Description copied from interface: ReplicationSourceInterface
Get the current log that's replicated

Specified by:
getCurrentPath in interface ReplicationSourceInterface
Returns:
the current log

setSourceEnabled

public void setSourceEnabled(boolean status)
Description copied from interface: ReplicationSourceInterface
Set if this source is enabled or disabled

Specified by:
setSourceEnabled in interface ReplicationSourceInterface
Parameters:
status - the new status


Copyright © 2011 The Apache Software Foundation. All Rights Reserved.