|
|||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.conf.Configured
org.apache.nutch.crawl.Generator
Generates a subset of a crawl db to fetch.
Nested Class Summary | |
static class |
Generator.DecreasingFloatComparator
|
static class |
Generator.HashComparator
Sort fetch lists by hash of URL. |
static class |
Generator.Selector
Selects entries due for fetch. |
static class |
Generator.SelectorEntry
|
static class |
Generator.SelectorInverseMapper
|
Field Summary | |
static org.apache.commons.logging.Log |
LOG
|
Constructor Summary | |
Generator(Configuration conf)
Construct a generator. |
Method Summary | |
Path |
generate(Path dbDir,
Path segments)
Generate fetchlists in a segment. |
Path |
generate(Path dbDir,
Path segments,
int numLists,
long topN,
long curTime)
Generate fetchlists in a segment. |
static String |
generateSegmentName()
|
static void |
main(String[] args)
Generate a fetchlist from the pagedb and linkdb |
Methods inherited from class org.apache.hadoop.conf.Configured |
getConf, setConf |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
public static final org.apache.commons.logging.Log LOG
Constructor Detail |
public Generator(Configuration conf)
Method Detail |
public Path generate(Path dbDir, Path segments) throws IOException
IOException
public Path generate(Path dbDir, Path segments, int numLists, long topN, long curTime) throws IOException
IOException
public static String generateSegmentName()
public static void main(String[] args) throws Exception
Exception
|
|||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |