org.apache.hadoop.hbase.regionserver.wal
Class HLogSplitter

java.lang.Object
  extended by org.apache.hadoop.hbase.regionserver.wal.HLogSplitter

@InterfaceAudience.Private
public class HLogSplitter
extends Object

This class is responsible for splitting up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.


Field Summary
protected  org.apache.hadoop.conf.Configuration conf
           
protected  org.apache.hadoop.fs.FileSystem fs
           
protected  org.apache.hadoop.fs.Path oldLogDir
           
protected  org.apache.hadoop.fs.Path rootDir
           
protected  LastSequenceId sequenceIdChecker
           
protected  org.apache.hadoop.fs.Path srcDir
           
protected  AtomicReference<Throwable> thrown
           
 
Constructor Summary
HLogSplitter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path srcDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs, LastSequenceId idChecker)
           
 
Method Summary
static HLogSplitter createLogSplitter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path srcDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs)
          Create a new HLogSplitter using the given Configuration and the hbase.hlog.splitter.impl property to derived the instance class to use.
protected  HLog.Writer createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path logfile, org.apache.hadoop.conf.Configuration conf)
          Create a new HLog.Writer for writing log splits.
static void finishSplitLogFile(org.apache.hadoop.fs.Path rootdir, org.apache.hadoop.fs.Path oldLogDir, String logfile, org.apache.hadoop.conf.Configuration conf)
           
static void finishSplitLogFile(String logfile, org.apache.hadoop.conf.Configuration conf)
          Completes the work done by splitLogFile by archiving logs
protected  HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus file, org.apache.hadoop.conf.Configuration conf, boolean skipErrors)
          Create a new HLog.Reader for reading logs to split.
protected  HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path curLogFile, org.apache.hadoop.conf.Configuration conf)
          Create a new HLog.Reader for reading logs to split.
 long getSize()
           
 long getTime()
           
 List<org.apache.hadoop.fs.Path> splitLog()
          Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup.
 List<org.apache.hadoop.fs.Path> splitLog(CountDownLatch latch)
          Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup.
 boolean splitLogFile(org.apache.hadoop.fs.FileStatus logfile, CancelableProgressable reporter)
           
static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter)
          Splits a HLog file into region's recovered-edits directory
static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker)
          Splits a HLog file into region's recovered-edits directory
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

rootDir

protected final org.apache.hadoop.fs.Path rootDir

srcDir

protected final org.apache.hadoop.fs.Path srcDir

oldLogDir

protected final org.apache.hadoop.fs.Path oldLogDir

fs

protected final org.apache.hadoop.fs.FileSystem fs

conf

protected final org.apache.hadoop.conf.Configuration conf

thrown

protected AtomicReference<Throwable> thrown

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker
Constructor Detail

HLogSplitter

public HLogSplitter(org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.Path rootDir,
                    org.apache.hadoop.fs.Path srcDir,
                    org.apache.hadoop.fs.Path oldLogDir,
                    org.apache.hadoop.fs.FileSystem fs,
                    LastSequenceId idChecker)
Method Detail

createLogSplitter

public static HLogSplitter createLogSplitter(org.apache.hadoop.conf.Configuration conf,
                                             org.apache.hadoop.fs.Path rootDir,
                                             org.apache.hadoop.fs.Path srcDir,
                                             org.apache.hadoop.fs.Path oldLogDir,
                                             org.apache.hadoop.fs.FileSystem fs)
Create a new HLogSplitter using the given Configuration and the hbase.hlog.splitter.impl property to derived the instance class to use.

Parameters:
conf -
rootDir - hbase directory
srcDir - logs directory
oldLogDir - directory where processed logs are archived to
fs - FileSystem
Returns:
New HLogSplitter instance

splitLog

public List<org.apache.hadoop.fs.Path> splitLog()
                                         throws IOException
Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.

Returns:
the list of splits
Throws:
IOException - will throw if corrupted hlogs aren't tolerated

splitLog

public List<org.apache.hadoop.fs.Path> splitLog(CountDownLatch latch)
                                         throws IOException
Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.

Parameters:
latch -
Returns:
the list of splits
Throws:
IOException - will throw if corrupted hlogs aren't tolerated

getTime

public long getTime()
Returns:
time that this split took

getSize

public long getSize()
Returns:
aggregate size of hlogs that were split

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter,
                                   LastSequenceId idChecker)
                            throws IOException
Splits a HLog file into region's recovered-edits directory

If the log file has N regions then N recovered.edits files will be produced.

Parameters:
rootDir -
logfile -
fs -
conf -
reporter -
idChecker -
Returns:
false if it is interrupted by the progress-able.
Throws:
IOException

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter)
                            throws IOException
Splits a HLog file into region's recovered-edits directory

If the log file has N regions then N recovered.edits files will be produced.

Parameters:
rootDir -
logfile -
fs -
conf -
reporter -
Returns:
false if it is interrupted by the progress-able.
Throws:
IOException

splitLogFile

public boolean splitLogFile(org.apache.hadoop.fs.FileStatus logfile,
                            CancelableProgressable reporter)
                     throws IOException
Throws:
IOException

finishSplitLogFile

public static void finishSplitLogFile(String logfile,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException
Completes the work done by splitLogFile by archiving logs

It is invoked by SplitLogManager once it knows that one of the SplitLogWorkers have completed the splitLogFile() part. If the master crashes then this function might get called multiple times.

Parameters:
logfile -
conf -
Throws:
IOException

finishSplitLogFile

public static void finishSplitLogFile(org.apache.hadoop.fs.Path rootdir,
                                      org.apache.hadoop.fs.Path oldLogDir,
                                      String logfile,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException
Throws:
IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.FileStatus file,
                                org.apache.hadoop.conf.Configuration conf,
                                boolean skipErrors)
                         throws IOException,
                                org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException
Create a new HLog.Reader for reading logs to split.

Parameters:
fs -
file -
conf -
Returns:
A new Reader instance
Throws:
IOException
CorruptedLogFileException
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.CorruptedLogFileException

createWriter

protected HLog.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.fs.Path logfile,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws IOException
Create a new HLog.Writer for writing log splits.

Throws:
IOException

getReader

protected HLog.Reader getReader(org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path curLogFile,
                                org.apache.hadoop.conf.Configuration conf)
                         throws IOException
Create a new HLog.Reader for reading logs to split.

Throws:
IOException


Copyright © 2013 The Apache Software Foundation. All Rights Reserved.