org.apache.falcon.latedata
Class LateDataHandler
java.lang.Object
Configured
org.apache.falcon.latedata.LateDataHandler
public class LateDataHandler
- extends Configured
A tool for late data handling.
Method Summary |
long |
computeStorageMetric(String feedUriTemplate,
String feedStorageType,
Configuration conf)
This method computes the storage metrics for a given feed's instance or partition. |
String |
detectChanges(Path file,
Map<String,Long> metrics,
Configuration conf)
This method compares the recorded metrics persisted in file against
the recently computed metrics and returns the list of feeds that has changed. |
static void |
main(String[] args)
|
int |
run(String[] args)
|
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
LateDataHandler
public LateDataHandler()
main
public static void main(String[] args)
throws Exception
- Throws:
Exception
run
public int run(String[] args)
throws Exception
- Throws:
Exception
computeStorageMetric
public long computeStorageMetric(String feedUriTemplate,
String feedStorageType,
Configuration conf)
throws IOException,
FalconException,
URISyntaxException
- This method computes the storage metrics for a given feed's instance or partition.
It uses size on disk as the metric for File System Storage.
It uses create time as the metric for Catalog Table Storage.
The assumption is that if a partition has changed or reinstated, the underlying
metric would change, either size or create time.
- Parameters:
feedUriTemplate
- URI for the feed storage, filesystem path or table urifeedStorageType
- feed storage typeconf
- configuration
- Returns:
- computed metric
- Throws:
IOException
FalconException
URISyntaxException
detectChanges
public String detectChanges(Path file,
Map<String,Long> metrics,
Configuration conf)
throws Exception
- This method compares the recorded metrics persisted in file against
the recently computed metrics and returns the list of feeds that has changed.
- Parameters:
file
- persisted metrics from the first runmetrics
- newly computed metricsconf
- configuration
- Returns:
- list if feed names which has changed, empty string is none has changed
- Throws:
Exception
Copyright © 2013-2014 Apache Software Foundation. All Rights Reserved.