org.apache.hadoop.chukwa.extraction.archive
Class SinkArchiver

java.lang.Object
  extended by org.apache.hadoop.chukwa.extraction.archive.SinkArchiver
All Implemented Interfaces:
CHUKWA_CONSTANT

public class SinkArchiver
extends Object
implements CHUKWA_CONSTANT

A lightweight tool for archiving, suitable for small-to-medium-size Chukwa deployments that don't use Demux. Grabs everything in the data sink, runs the Archiver MapReduce job, then promotes output to the archive dir. Input is determined by conf option chukwaArchiveDir; defaults to /chukwa/logs Uses /chukwa/archivesProcessing/mr[Input/Output] as tmp storage Outputs to /chukwa/archives


Field Summary
static org.apache.hadoop.fs.PathFilter DATA_SINK_FILTER
           
 
Fields inherited from interface org.apache.hadoop.chukwa.extraction.CHUKWA_CONSTANT
ARCHIVES_IN_ERROR_DIR_NAME, ARCHIVES_MR_INPUT_DIR_NAME, ARCHIVES_MR_OUTPUT_DIR_NAME, ARCHIVES_PROCESSING_DIR_NAME, CHUKWA_ARCHIVE_DIR_FIELD, CHUKWA_DATA_SINK_DIR_FIELD, CHUKWA_DEMUX_REDUCER_COUNT_FIELD, CHUKWA_NAGIOS_HOST_FIELD, CHUKWA_NAGIOS_PORT_FIELD, CHUKWA_POST_PROCESS_DIR_FIELD, CHUKWA_POSTPROCESS_IN_ERROR_DIR_FIELD, CHUKWA_REPORTING_HOST_FIELD, CHUKWA_ROOT_DIR_FIELD, CHUKWA_ROOT_REPOS_DIR_FIELD, DEFAULT_CHUKWA_DATASINK_DIR_NAME, DEFAULT_CHUKWA_LOGS_DIR_NAME, DEFAULT_CHUKWA_POSTPROCESS_DIR_NAME, DEFAULT_CHUKWA_ROOT_DIR_NAME, DEFAULT_DEMUX_IN_ERROR_DIR_NAME, DEFAULT_DEMUX_MR_INPUT_DIR_NAME, DEFAULT_DEMUX_MR_OUTPUT_DIR_NAME, DEFAULT_DEMUX_PROCESSING_DIR_NAME, DEFAULT_FINAL_ARCHIVES, DEFAULT_POSTPROCESS_IN_ERROR_DIR_NAME, DEFAULT_REPOS_DIR_NAME, HDFS_DEFAULT_NAME_FIELD, POST_DEMUX_DATA_LOADER, WRITER_HDFS_FILESYSTEM_FIELD
 
Constructor Summary
SinkArchiver()
           
 
Method Summary
 void exec(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf)
           
static void main(String[] args)
           
 void promoteAndMerge(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dest)
          Merges the contents of src into dest.
 int runMapRedJob(org.apache.hadoop.conf.Configuration conf, String in, String out)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

DATA_SINK_FILTER

public static final org.apache.hadoop.fs.PathFilter DATA_SINK_FILTER
Constructor Detail

SinkArchiver

public SinkArchiver()
Method Detail

main

public static void main(String[] args)

exec

public void exec(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.conf.Configuration conf)

runMapRedJob

public int runMapRedJob(org.apache.hadoop.conf.Configuration conf,
                        String in,
                        String out)
                 throws Exception
Throws:
Exception

promoteAndMerge

public void promoteAndMerge(org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path src,
                            org.apache.hadoop.fs.Path dest)
                     throws IOException
Merges the contents of src into dest. If src is a file, moves it to dest.

Parameters:
fs - the filesystem in question
src - a file or directory to merge into dest
dest - a directory to merge into
Throws:
IOException


Copyright © ${year} The Apache Software Foundation