org.apache.hadoop.hbase.replication.regionserver
Class ReplicationSourceManager

java.lang.Object
  extended by org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager
All Implemented Interfaces:
ReplicationListener

@InterfaceAudience.Private
public class ReplicationSourceManager
extends Object
implements ReplicationListener

This class is responsible to manage all the replication sources. There are two classes of sources:

  • Normal sources are persistent and one per peer cluster
  • Old sources are recovered from a failed region server and our only goal is to finish replicating the HLog queue it had up in ZK
  • When a region server dies, this class uses a watcher to get notified and it tries to grab a lock in order to transfer all the queues in a local old source. This class implements the ReplicationListener interface so that it can track changes in replication state.


    Constructor Summary
    ReplicationSourceManager(ReplicationQueues replicationQueues, ReplicationPeers replicationPeers, ReplicationTracker replicationTracker, org.apache.hadoop.conf.Configuration conf, Server server, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path logDir, org.apache.hadoop.fs.Path oldLogDir, UUID clusterId)
              Creates a replication manager and sets the watch on all the other registered region servers
     
    Method Summary
    protected  ReplicationSourceInterface addSource(String id)
              Add a new normal source to this region server
     void cleanOldLogs(String key, String id, boolean queueRecovered)
              Cleans a log file and all older files from ZK.
     void closeRecoveredQueue(ReplicationSourceInterface src)
              Clear the references to the specified old source
     void deleteSource(String peerId, boolean closeConnection)
              Delete a complete queue of hlogs associated with a peer cluster
     org.apache.hadoop.fs.FileSystem getFs()
              Get the handle on the local file system
    protected  Map<String,SortedSet<String>> getHLogs()
              Get a copy of the hlogs of the first source on this rs
    protected  Map<String,SortedSet<String>> getHlogsByIdRecoveredQueues()
              Get a copy of the hlogs of the recovered sources on this rs
     org.apache.hadoop.fs.Path getLogDir()
              Get the directory where hlogs are stored by their RSs
     org.apache.hadoop.fs.Path getOldLogDir()
              Get the directory where hlogs are archived
     List<ReplicationSourceInterface> getOldSources()
              Get a list of all the old sources of this rs
    protected  ReplicationSourceInterface getReplicationSource(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, ReplicationSourceManager manager, ReplicationQueues replicationQueues, ReplicationPeers replicationPeers, Server server, String peerId, UUID clusterId, ReplicationPeerConfig peerConfig, ReplicationPeer replicationPeer)
              Factory method to create a replication source
     List<ReplicationSourceInterface> getSources()
              Get a list of all the normal sources of this rs
     String getStats()
              Get a string representation of all the sources' metrics
    protected  void init()
              Adds a normal source per registered peer cluster and tries to process all old region server hlog queues
     void join()
              Terminate the replication on this region server
     void logPositionAndCleanOldLogs(org.apache.hadoop.fs.Path log, String id, long position, boolean queueRecovered, boolean holdLogInZK)
              Provide the id of the peer and a log key and this method will figure which hlog it belongs to and will log, for this region server, the current position.
     void peerListChanged(List<String> peerIds)
              The list of registered peer clusters has changed.
     void peerRemoved(String peerId)
              A peer cluster has been removed (i.e.
     void regionServerRemoved(String regionserver)
              A region server has been removed from the local cluster
     void removePeer(String id)
              Thie method first deletes all the recovered sources for the specified id, then deletes the normal source (deleting all related data in ZK).
     
    Methods inherited from class java.lang.Object
    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
     

    Constructor Detail

    ReplicationSourceManager

    public ReplicationSourceManager(ReplicationQueues replicationQueues,
                                    ReplicationPeers replicationPeers,
                                    ReplicationTracker replicationTracker,
                                    org.apache.hadoop.conf.Configuration conf,
                                    Server server,
                                    org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.Path logDir,
                                    org.apache.hadoop.fs.Path oldLogDir,
                                    UUID clusterId)
    Creates a replication manager and sets the watch on all the other registered region servers

    Parameters:
    replicationQueues - the interface for manipulating replication queues
    replicationPeers -
    replicationTracker -
    conf - the configuration to use
    stopper - the stopper object for this region server
    fs - the file system to use
    logDir - the directory that contains all hlog directories of live RSs
    oldLogDir - the directory where old logs are archived
    clusterId -
    Method Detail

    logPositionAndCleanOldLogs

    public void logPositionAndCleanOldLogs(org.apache.hadoop.fs.Path log,
                                           String id,
                                           long position,
                                           boolean queueRecovered,
                                           boolean holdLogInZK)
    Provide the id of the peer and a log key and this method will figure which hlog it belongs to and will log, for this region server, the current position. It will also clean old logs from the queue.

    Parameters:
    log - Path to the log currently being replicated from replication status in zookeeper. It will also delete older entries.
    id - id of the peer cluster
    position - current location in the log
    queueRecovered - indicates if this queue comes from another region server
    holdLogInZK - if true then the log is retained in ZK

    cleanOldLogs

    public void cleanOldLogs(String key,
                             String id,
                             boolean queueRecovered)
    Cleans a log file and all older files from ZK. Called when we are sure that a log file is closed and has no more entries.

    Parameters:
    key - Path to the log
    id - id of the peer cluster
    queueRecovered - Whether this is a recovered queue

    init

    protected void init()
                 throws IOException,
                        ReplicationException
    Adds a normal source per registered peer cluster and tries to process all old region server hlog queues

    Throws:
    IOException
    ReplicationException

    addSource

    protected ReplicationSourceInterface addSource(String id)
                                            throws IOException,
                                                   ReplicationException
    Add a new normal source to this region server

    Parameters:
    id - the id of the peer cluster
    Returns:
    the source that was created
    Throws:
    IOException
    ReplicationException

    deleteSource

    public void deleteSource(String peerId,
                             boolean closeConnection)
    Delete a complete queue of hlogs associated with a peer cluster

    Parameters:
    peerId - Id of the peer cluster queue of hlogs to delete

    join

    public void join()
    Terminate the replication on this region server


    getHLogs

    protected Map<String,SortedSet<String>> getHLogs()
    Get a copy of the hlogs of the first source on this rs

    Returns:
    a sorted set of hlog names

    getHlogsByIdRecoveredQueues

    protected Map<String,SortedSet<String>> getHlogsByIdRecoveredQueues()
    Get a copy of the hlogs of the recovered sources on this rs

    Returns:
    a sorted set of hlog names

    getSources

    public List<ReplicationSourceInterface> getSources()
    Get a list of all the normal sources of this rs

    Returns:
    lis of all sources

    getOldSources

    public List<ReplicationSourceInterface> getOldSources()
    Get a list of all the old sources of this rs

    Returns:
    list of all old sources

    getReplicationSource

    protected ReplicationSourceInterface getReplicationSource(org.apache.hadoop.conf.Configuration conf,
                                                              org.apache.hadoop.fs.FileSystem fs,
                                                              ReplicationSourceManager manager,
                                                              ReplicationQueues replicationQueues,
                                                              ReplicationPeers replicationPeers,
                                                              Server server,
                                                              String peerId,
                                                              UUID clusterId,
                                                              ReplicationPeerConfig peerConfig,
                                                              ReplicationPeer replicationPeer)
                                                       throws IOException
    Factory method to create a replication source

    Parameters:
    conf - the configuration to use
    fs - the file system to use
    manager - the manager to use
    stopper - the stopper object for this region server
    peerId - the id of the peer cluster
    Returns:
    the created source
    Throws:
    IOException

    closeRecoveredQueue

    public void closeRecoveredQueue(ReplicationSourceInterface src)
    Clear the references to the specified old source

    Parameters:
    src - source to clear

    removePeer

    public void removePeer(String id)
    Thie method first deletes all the recovered sources for the specified id, then deletes the normal source (deleting all related data in ZK).

    Parameters:
    id - The id of the peer cluster

    regionServerRemoved

    public void regionServerRemoved(String regionserver)
    Description copied from interface: ReplicationListener
    A region server has been removed from the local cluster

    Specified by:
    regionServerRemoved in interface ReplicationListener
    Parameters:
    regionserver - the removed region server

    peerRemoved

    public void peerRemoved(String peerId)
    Description copied from interface: ReplicationListener
    A peer cluster has been removed (i.e. unregistered) from replication.

    Specified by:
    peerRemoved in interface ReplicationListener
    Parameters:
    peerId - The peer id of the cluster that has been removed

    peerListChanged

    public void peerListChanged(List<String> peerIds)
    Description copied from interface: ReplicationListener
    The list of registered peer clusters has changed.

    Specified by:
    peerListChanged in interface ReplicationListener
    Parameters:
    peerIds - A list of all currently registered peer clusters

    getOldLogDir

    public org.apache.hadoop.fs.Path getOldLogDir()
    Get the directory where hlogs are archived

    Returns:
    the directory where hlogs are archived

    getLogDir

    public org.apache.hadoop.fs.Path getLogDir()
    Get the directory where hlogs are stored by their RSs

    Returns:
    the directory where hlogs are stored by their RSs

    getFs

    public org.apache.hadoop.fs.FileSystem getFs()
    Get the handle on the local file system

    Returns:
    Handle on the local file system

    getStats

    public String getStats()
    Get a string representation of all the sources' metrics



    Copyright © 2007–2016 The Apache Software Foundation. All rights reserved.