org.apache.hadoop.hbase.regionserver
Class HLog

java.lang.Object
  extended by org.apache.hadoop.hbase.regionserver.HLog
All Implemented Interfaces:
org.apache.hadoop.fs.Syncable, HConstants

public class HLog
extends Object
implements HConstants, org.apache.hadoop.fs.Syncable

HLog stores all the edits to the HStore. It performs logfile-rolling, so external callers are not aware that the underlying file is being rolled.

A single HLog is used by several HRegions simultaneously.

Each HRegion is identified by a unique long int. HRegions do not need to declare themselves before using the HLog; they simply include their HRegion-id in the append or completeCacheFlush calls.

An HLog consists of multiple on-disk files, which have a chronological order. As data is flushed to other (better) on-disk structures, the log becomes obsolete. We can destroy all the log messages for a given HRegion-id up to the most-recent CACHEFLUSH message from that HRegion.

It's only practical to delete entire files. Thus, we delete an entire on-disk file F when all of the messages in F have a log-sequence-id that's older (smaller) than the most-recent CACHEFLUSH message for every HRegion that has a message in F.

Synchronized methods can never execute in parallel. However, between the start of a cache flush and the completion point, appends are allowed but log rolling is not. To prevent log rolling taking place during this period, a separate reentrant lock is used.


Field Summary
 
Fields inherited from interface org.apache.hadoop.hbase.HConstants
ALL_META_COLUMNS, ALL_VERSIONS, COL_REGIONINFO, COL_REGIONINFO_ARRAY, COL_SERVER, COL_SPLITA, COL_SPLITB, COL_STARTCODE, COLUMN_FAMILY, COLUMN_FAMILY_ARRAY, COLUMN_FAMILY_HISTORIAN, COLUMN_FAMILY_HISTORIAN_STR, COLUMN_FAMILY_STR, DEFAULT_CLIENT_RETRIES, DEFAULT_HOST, DEFAULT_MASTER_ADDRESS, DEFAULT_MASTER_INFOPORT, DEFAULT_MASTER_PORT, DEFAULT_MAX_FILE_SIZE, DEFAULT_REGION_SERVER_CLASS, DEFAULT_REGIONSERVER_ADDRESS, DEFAULT_REGIONSERVER_INFOPORT, DEFAULT_SIZE_RESERVATION_BLOCK, EMPTY_BYTE_ARRAY, EMPTY_END_ROW, EMPTY_START_ROW, FILE_SYSTEM_VERSION, FOREVER, HBASE_CLIENT_RETRIES_NUMBER_KEY, HBASE_DIR, HREGION_LOGDIR_NAME, HREGION_OLDLOGFILE_NAME, IN_MEMORY, LAST_ROW, LATEST_TIMESTAMP, MAJOR_COMPACTION_PERIOD, MASTER_ADDRESS, MAX_ROW_LENGTH, META_ROW_DELIMITER, META_TABLE_NAME, MODIFY_CLOSE_REGION, MODIFY_TABLE_COMPACT, MODIFY_TABLE_FLUSH, MODIFY_TABLE_MAJOR_COMPACT, MODIFY_TABLE_SET_HTD, MODIFY_TABLE_SPLIT, NAME, NINES, REGION_SERVER_CLASS, REGION_SERVER_IMPL, REGIONSERVER_ADDRESS, RETRY_BACKOFF, ROOT_TABLE_NAME, THREAD_WAKE_FREQUENCY, UTF8_ENCODING, VERSION_FILE_NAME, VERSIONS, WEEK_IN_SECONDS, ZERO_L, ZEROES
 
Constructor Summary
HLog(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, org.apache.hadoop.conf.Configuration conf, LogRollListener listener)
          Create an edit log at the given dir location.
 
Method Summary
 void append(HRegionInfo regionInfo, byte[] row, HLogEdit logEdit)
          Append an entry to the log.
 void append(HRegionInfo regionInfo, HLogEdit logEdit)
          Append an entry without a row to the log.
 void close()
          Shut down the log.
 void closeAndDelete()
          Shut down the log and delete the log directory
 org.apache.hadoop.fs.Path computeFilename(long fn)
          This is a convenience method that computes a new filename with a given file-number.
 long getFilenum()
           
static String getHLogDirectoryName(HServerInfo info)
          Construct the HLog directory name
 long getSequenceNumber()
           
static boolean isMetaColumn(byte[] column)
           
static void main(String[] args)
          Pass one or more log file names and it will either dump out a text version on stdout or split the specified log files.
 byte[] rollWriter()
          Roll the log writer.
static void splitLog(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path srcDir, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
          Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup.
 void sync()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

HLog

public HLog(org.apache.hadoop.fs.FileSystem fs,
            org.apache.hadoop.fs.Path dir,
            org.apache.hadoop.conf.Configuration conf,
            LogRollListener listener)
     throws IOException
Create an edit log at the given dir location. You should never have to load an existing log. If there is a log at startup, it should have already been processed and deleted by the time the HLog object is started up.

Parameters:
fs -
dir -
conf -
listener -
Throws:
IOException
Method Detail

getFilenum

public long getFilenum()

getSequenceNumber

public long getSequenceNumber()
Returns:
log sequence number

rollWriter

public byte[] rollWriter()
                  throws org.apache.hadoop.hbase.regionserver.FailedLogCloseException,
                         IOException
Roll the log writer. That is, start writing log messages to a new file. Because a log cannot be rolled during a cache flush, and a cache flush spans two method calls, a special lock needs to be obtained so that a cache flush cannot start when the log is being rolled and the log cannot be rolled during a cache flush.

Note that this method cannot be synchronized because it is possible that startCacheFlush runs, obtaining the cacheFlushLock, then this method could start which would obtain the lock on this but block on obtaining the cacheFlushLock and then completeCacheFlush could be called which would wait for the lock on this and consequently never release the cacheFlushLock

Returns:
If lots of logs, flush the returned region so next time through we can clean logs. Returns null if nothing to flush.
Throws:
FailedLogCloseException
IOException

computeFilename

public org.apache.hadoop.fs.Path computeFilename(long fn)
This is a convenience method that computes a new filename with a given file-number.

Parameters:
fn -
Returns:
Path

closeAndDelete

public void closeAndDelete()
                    throws IOException
Shut down the log and delete the log directory

Throws:
IOException

close

public void close()
           throws IOException
Shut down the log.

Throws:
IOException

sync

public void sync()
          throws IOException
Specified by:
sync in interface org.apache.hadoop.fs.Syncable
Throws:
IOException

append

public void append(HRegionInfo regionInfo,
                   HLogEdit logEdit)
            throws IOException
Append an entry without a row to the log.

Parameters:
regionInfo -
logEdit -
Throws:
IOException

append

public void append(HRegionInfo regionInfo,
                   byte[] row,
                   HLogEdit logEdit)
            throws IOException
Append an entry to the log.

Parameters:
regionInfo -
row -
logEdit -
Throws:
IOException

isMetaColumn

public static boolean isMetaColumn(byte[] column)
Parameters:
column -
Returns:
true if the column is a meta column

splitLog

public static void splitLog(org.apache.hadoop.fs.Path rootDir,
                            org.apache.hadoop.fs.Path srcDir,
                            org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.conf.Configuration conf)
                     throws IOException
Split up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region for region to replay on startup. Delete the old log files when finished.

Parameters:
rootDir - qualified root directory of the HBase instance
srcDir - Directory of log files to split: e.g. ${ROOTDIR}/log_HOST_PORT
fs - FileSystem
conf - HBaseConfiguration
Throws:
IOException

getHLogDirectoryName

public static String getHLogDirectoryName(HServerInfo info)
Construct the HLog directory name

Parameters:
info - HServerInfo for server
Returns:
the HLog directory name

main

public static void main(String[] args)
                 throws IOException
Pass one or more log file names and it will either dump out a text version on stdout or split the specified log files.

Parameters:
args -
Throws:
IOException


Copyright © 2008 The Apache Software Foundation