org.apache.hadoop.hbase.regionserver.wal
Class HLogSplitter

java.lang.Object
  extended by org.apache.hadoop.hbase.regionserver.wal.HLogSplitter

@InterfaceAudience.Private
public class HLogSplitter
extends Object

This class is responsible for splitting up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.


Nested Class Summary
static class HLogSplitter.MutationReplay
          A struct used by getMutationsFromWALEntry
 
Field Summary
protected  org.apache.hadoop.conf.Configuration conf
           
protected  boolean distributedLogReplay
           
protected  String failedServerName
           
protected  org.apache.hadoop.fs.FileSystem fs
           
protected  Map<String,Long> lastFlushedSequenceIds
           
protected  Map<String,Map<byte[],Long>> regionMaxSeqIdInStores
           
protected  org.apache.hadoop.fs.Path rootDir
           
protected  LastSequenceId sequenceIdChecker
           
protected  AtomicReference<Throwable> thrown
           
 
Method Summary
protected  HLog.Writer createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path logfile, org.apache.hadoop.conf.Configuration conf)
          Create a new HLog.Writer for writing log splits.
static void finishSplitLogFile(String logfile, org.apache.hadoop.conf.Configuration conf)
          Completes the work done by splitLogFile by archiving logs
static List<HLogSplitter.MutationReplay> getMutationsFromWALEntry(AdminProtos.WALEntry entry, CellScanner cells, Pair<HLogKey,WALEdit> logEntry, boolean addLogReplayTag)
          This function is used to construct mutations from a WALEntry.
protected  HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus file, org.apache.hadoop.conf.Configuration conf, boolean skipErrors, CancelableProgressable reporter)
          Create a new HLog.Reader for reading logs to split.
protected  HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path curLogFile, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter)
          Create a new HLog.Reader for reading logs to split.
static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path logDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
           
static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, ZooKeeperWatcher zkw, ZooKeeperProtos.SplitLogTask.RecoveryMode mode)
          Splits a HLog file into region's recovered-edits directory.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

rootDir

protected final org.apache.hadoop.fs.Path rootDir

fs

protected final org.apache.hadoop.fs.FileSystem fs

conf

protected final org.apache.hadoop.conf.Configuration conf

thrown

protected AtomicReference<Throwable> thrown

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

distributedLogReplay

protected boolean distributedLogReplay

lastFlushedSequenceIds

protected Map<String,Long> lastFlushedSequenceIds

regionMaxSeqIdInStores

protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

failedServerName

protected String failedServerName
Method Detail

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter,
                                   LastSequenceId idChecker,
                                   ZooKeeperWatcher zkw,
                                   ZooKeeperProtos.SplitLogTask.RecoveryMode mode)
                            throws IOException
Splits a HLog file into region's recovered-edits directory. This is the main entry point for distributed log splitting from SplitLogWorker.

If the log file has N regions then N recovered.edits files will be produced.

Parameters:
rootDir -
logfile -
fs -
conf -
reporter -
idChecker -
zkw - ZooKeeperWatcher if it's null, we will back to the old-style log splitting where we dump out recoved.edits files for regions to replay on.
Returns:
false if it is interrupted by the progress-able.
Throws:
IOException

split

public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path rootDir,
                                                    org.apache.hadoop.fs.Path logDir,
                                                    org.apache.hadoop.fs.Path oldLogDir,
                                                    org.apache.hadoop.fs.FileSystem fs,
                                                    org.apache.hadoop.conf.Configuration conf)
                                             throws IOException
Throws:
IOException

finishSplitLogFile

public static void finishSplitLogFile(String logfile,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException
Completes the work done by splitLogFile by archiving logs

It is invoked by SplitLogManager once it knows that one of the SplitLogWorkers have completed the splitLogFile() part. If the master crashes then this function might get called multiple times.

Parameters:
logfile -
conf -
Throws:
IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.FileStatus file,
                                org.apache.hadoop.conf.Configuration conf,
                                boolean skipErrors,
                                CancelableProgressable reporter)
                         throws IOException,
                                org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException
Create a new HLog.Reader for reading logs to split.

Parameters:
fs -
file -
conf -
Returns:
A new Reader instance
Throws:
IOException
CorruptedLogFileException
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException

createWriter

protected HLog.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.fs.Path logfile,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws IOException
Create a new HLog.Writer for writing log splits.

Throws:
IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path curLogFile,
                                org.apache.hadoop.conf.Configuration conf,
                                CancelableProgressable reporter)
                         throws IOException
Create a new HLog.Reader for reading logs to split.

Throws:
IOException

getMutationsFromWALEntry

public static List<HLogSplitter.MutationReplay> getMutationsFromWALEntry(AdminProtos.WALEntry entry,
                                                                         CellScanner cells,
                                                                         Pair<HLogKey,WALEdit> logEntry,
                                                                         boolean addLogReplayTag)
                                                                  throws IOException
This function is used to construct mutations from a WALEntry. It also reconstructs HLogKey & WALEdit from the passed in WALEntry

Parameters:
entry -
cells -
logEntry - pair of HLogKey and WALEdit instance stores HLogKey and WALEdit instances extracted from the passed in WALEntry.
addLogReplayTag -
Returns:
list of Pair to be replayed
Throws:
IOException


Copyright © 2015 The Apache Software Foundation. All rights reserved.