org.apache.hadoop.chukwa.extraction.archive
Class SinkArchiver
java.lang.Object
org.apache.hadoop.chukwa.extraction.archive.SinkArchiver
- All Implemented Interfaces:
- CHUKWA_CONSTANT
public class SinkArchiver
- extends Object
- implements CHUKWA_CONSTANT
A lightweight tool for archiving, suitable for small-to-medium-size Chukwa
deployments that don't use Demux.
Grabs everything in the data sink, runs the Archiver MapReduce job,
then promotes output to the archive dir.
Input is determined by conf option chukwaArchiveDir; defaults to
/chukwa/logs
Uses /chukwa/archivesProcessing/mr[Input/Output] as tmp storage
Outputs to /chukwa/archives
Fields inherited from interface org.apache.hadoop.chukwa.extraction.CHUKWA_CONSTANT |
ARCHIVES_IN_ERROR_DIR_NAME, ARCHIVES_MR_INPUT_DIR_NAME, ARCHIVES_MR_OUTPUT_DIR_NAME, ARCHIVES_PROCESSING_DIR_NAME, CHUKWA_ARCHIVE_DIR_FIELD, CHUKWA_DATA_SINK_DIR_FIELD, CHUKWA_DEMUX_REDUCER_COUNT_FIELD, CHUKWA_NAGIOS_HOST_FIELD, CHUKWA_NAGIOS_PORT_FIELD, CHUKWA_POST_PROCESS_DIR_FIELD, CHUKWA_POSTPROCESS_IN_ERROR_DIR_FIELD, CHUKWA_REPORTING_HOST_FIELD, CHUKWA_ROOT_DIR_FIELD, CHUKWA_ROOT_REPOS_DIR_FIELD, DEFAULT_CHUKWA_DATASINK_DIR_NAME, DEFAULT_CHUKWA_LOGS_DIR_NAME, DEFAULT_CHUKWA_POSTPROCESS_DIR_NAME, DEFAULT_CHUKWA_ROOT_DIR_NAME, DEFAULT_DEMUX_IN_ERROR_DIR_NAME, DEFAULT_DEMUX_MR_INPUT_DIR_NAME, DEFAULT_DEMUX_MR_OUTPUT_DIR_NAME, DEFAULT_DEMUX_PROCESSING_DIR_NAME, DEFAULT_FINAL_ARCHIVES, DEFAULT_POSTPROCESS_IN_ERROR_DIR_NAME, DEFAULT_REPOS_DIR_NAME, POST_DEMUX_DATA_LOADER, WRITER_HDFS_FILESYSTEM_FIELD |
Method Summary |
void |
exec(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.conf.Configuration conf)
|
static void |
main(String[] args)
|
void |
promoteAndMerge(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dest)
Merges the contents of src into dest. |
int |
runMapRedJob(org.apache.hadoop.conf.Configuration conf,
String in,
String out)
|
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
DATA_SINK_FILTER
public static final org.apache.hadoop.fs.PathFilter DATA_SINK_FILTER
SinkArchiver
public SinkArchiver()
main
public static void main(String[] args)
exec
public void exec(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.conf.Configuration conf)
runMapRedJob
public int runMapRedJob(org.apache.hadoop.conf.Configuration conf,
String in,
String out)
throws Exception
- Throws:
Exception
promoteAndMerge
public void promoteAndMerge(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dest)
throws IOException
- Merges the contents of src into dest.
If src is a file, moves it to dest.
- Parameters:
fs
- the filesystem in questionsrc
- a file or directory to merge into destdest
- a directory to merge into
- Throws:
IOException
Copyright © ${year} The Apache Software Foundation