org.apache.falcon.latedata
Class LateDataHandler

java.lang.Object
  extended by Configured
      extended by org.apache.falcon.latedata.LateDataHandler

public class LateDataHandler
extends Configured

A tool for late data handling.


Constructor Summary
LateDataHandler()
           
 
Method Summary
 long computeStorageMetric(String feedUriTemplate, String feedStorageType, Configuration conf)
          This method computes the storage metrics for a given feed's instance or partition.
 String detectChanges(Path file, Map<String,Long> metrics, Configuration conf)
          This method compares the recorded metrics persisted in file against the recently computed metrics and returns the list of feeds that has changed.
static void main(String[] args)
           
 int run(String[] args)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

LateDataHandler

public LateDataHandler()
Method Detail

main

public static void main(String[] args)
                 throws Exception
Throws:
Exception

run

public int run(String[] args)
        throws Exception
Throws:
Exception

computeStorageMetric

public long computeStorageMetric(String feedUriTemplate,
                                 String feedStorageType,
                                 Configuration conf)
                          throws IOException,
                                 FalconException,
                                 URISyntaxException
This method computes the storage metrics for a given feed's instance or partition. It uses size on disk as the metric for File System Storage. It uses create time as the metric for Catalog Table Storage. The assumption is that if a partition has changed or reinstated, the underlying metric would change, either size or create time.

Parameters:
feedUriTemplate - URI for the feed storage, filesystem path or table uri
feedStorageType - feed storage type
conf - configuration
Returns:
computed metric
Throws:
IOException
FalconException
URISyntaxException

detectChanges

public String detectChanges(Path file,
                            Map<String,Long> metrics,
                            Configuration conf)
                     throws Exception
This method compares the recorded metrics persisted in file against the recently computed metrics and returns the list of feeds that has changed.

Parameters:
file - persisted metrics from the first run
metrics - newly computed metrics
conf - configuration
Returns:
list if feed names which has changed, empty string is none has changed
Throws:
Exception


Copyright © 2013-2014 Apache Software Foundation. All Rights Reserved.