org.apache.hadoop.hbase.regionserver.compactions
Class DateTieredCompactionPolicy

java.lang.Object
  extended by org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy
      extended by org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy
          extended by org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy

@InterfaceAudience.LimitedPrivate(value="Configuration")
public class DateTieredCompactionPolicy
extends SortedCompactionPolicy

HBASE-15181 This is a simple implementation of date-based tiered compaction similar to Cassandra's for the following benefits: <<<<<<< HEAD 1. Improve date-range-based scan by structuring store files in date-based tiered layout. 2. Reduce compaction overhead. 3. Improve TTL efficiency. Perfect fit for the use cases that: 1. has mostly date-based data write and scan and a focus on the most recent data. Out-of-order writes are handled gracefully. Time range overlapping among store files is tolerated and the performance impact is minimized. Configuration can be set at hbase-site or overridden at per-table or per-column-family level by hbase shell. Design spec is at =======

  1. Improve date-range-based scan by structuring store files in date-based tiered layout.
  2. Reduce compaction overhead.
  3. Improve TTL efficiency.
Perfect fit for the use cases that:
  1. has mostly date-based data write and scan and a focus on the most recent data.
Out-of-order writes are handled gracefully. Time range overlapping among store files is tolerated and the performance impact is minimized. Configuration can be set at hbase-site or overridden at per-table or per-column-family level by hbase shell. Design spec is at >>>>>>> 3f749fd... HBASE-15368 Add pluggable window support https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/


Field Summary
 
Fields inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy
comConf, storeConfigInfo
 
Constructor Summary
DateTieredCompactionPolicy(org.apache.hadoop.conf.Configuration conf, StoreConfigInformation storeConfigInfo)
           
 
Method Summary
protected  CompactionRequest getCompactionRequest(ArrayList<StoreFile> candidateSelection, boolean tryingMajor, boolean isUserCompaction, boolean mayUseOffPeak, boolean mayBeStuck)
           
 boolean needsCompaction(Collection<StoreFile> storeFiles, List<StoreFile> filesCompacting)
          Heuristics for guessing whether we need minor compaction.
 CompactionRequest selectMajorCompaction(ArrayList<StoreFile> candidateSelection)
           
 CompactionRequest selectMinorCompaction(ArrayList<StoreFile> candidateSelection, boolean mayUseOffPeak, boolean mayBeStuck)
          We receive store files sorted in ascending order by seqId then scan the list of files.
 boolean shouldPerformMajorCompaction(Collection<StoreFile> filesToCompact)
           
 
Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy
checkMinFilesCriteria, filterBulk, getCurrentEligibleFiles, getNextMajorCompactTime, preSelectCompactionForCoprocessor, removeExcessFiles, selectCompaction, skipLargeFiles, throttleCompaction
 
Methods inherited from class org.apache.hadoop.hbase.regionserver.compactions.CompactionPolicy
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

DateTieredCompactionPolicy

public DateTieredCompactionPolicy(org.apache.hadoop.conf.Configuration conf,
                                  StoreConfigInformation storeConfigInfo)
                           throws IOException
Throws:
IOException
Method Detail

needsCompaction

public boolean needsCompaction(Collection<StoreFile> storeFiles,
                               List<StoreFile> filesCompacting)
Heuristics for guessing whether we need minor compaction.

Overrides:
needsCompaction in class SortedCompactionPolicy
Parameters:
storeFiles - files in the store.
filesCompacting - files being scheduled to compact.
Returns:
true to schedule a request.

shouldPerformMajorCompaction

public boolean shouldPerformMajorCompaction(Collection<StoreFile> filesToCompact)
                                     throws IOException
Specified by:
shouldPerformMajorCompaction in class SortedCompactionPolicy
Parameters:
filesToCompact - Files to compact. Can be null.
Returns:
True if we should run a major compaction.
Throws:
IOException

getCompactionRequest

protected CompactionRequest getCompactionRequest(ArrayList<StoreFile> candidateSelection,
                                                 boolean tryingMajor,
                                                 boolean isUserCompaction,
                                                 boolean mayUseOffPeak,
                                                 boolean mayBeStuck)
                                          throws IOException
Specified by:
getCompactionRequest in class SortedCompactionPolicy
Throws:
IOException

selectMajorCompaction

public CompactionRequest selectMajorCompaction(ArrayList<StoreFile> candidateSelection)

selectMinorCompaction

public CompactionRequest selectMinorCompaction(ArrayList<StoreFile> candidateSelection,
                                               boolean mayUseOffPeak,
                                               boolean mayBeStuck)
                                        throws IOException
We receive store files sorted in ascending order by seqId then scan the list of files. If the current file has a maxTimestamp older than last known maximum, treat this file as it carries the last known maximum. This way both seqId and timestamp are in the same order. If files carry the same maxTimestamps, they are ordered by seqId. We then reverse the list so they are ordered by seqId and maxTimestamp in descending order and build the time windows. All the out-of-order data into the same compaction windows, guaranteeing contiguous compaction based on sequence id.

Throws:
IOException


Copyright © 2007–2016 The Apache Software Foundation. All rights reserved.