org.apache.nutch.fetcher
Class Fetcher

java.lang.Object
  extended byorg.apache.hadoop.conf.Configured
      extended byorg.apache.nutch.fetcher.Fetcher
All Implemented Interfaces:
Configurable, JobConfigurable, MapRunnable

public class Fetcher
extends Configured
implements MapRunnable

The fetcher. Most of the work is done by plugins.


Nested Class Summary
static class Fetcher.InputFormat
           
 
Field Summary
static org.apache.commons.logging.Log LOG
           
static String SCORE_KEY
           
static String SEGMENT_NAME_KEY
           
static String SIGNATURE_KEY
           
 
Constructor Summary
Fetcher()
           
Fetcher(Configuration conf)
           
 
Method Summary
 void close()
           
 void configure(JobConf job)
           
 void fetch(Path segment, int threads, boolean parsing)
           
static boolean isParsing(Configuration conf)
           
static boolean isStoringContent(Configuration conf)
           
static void main(String[] args)
          Run the fetcher.
 void run(RecordReader input, OutputCollector output, Reporter reporter)
           
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

SIGNATURE_KEY

public static final String SIGNATURE_KEY
See Also:
Constant Field Values

SEGMENT_NAME_KEY

public static final String SEGMENT_NAME_KEY
See Also:
Constant Field Values

SCORE_KEY

public static final String SCORE_KEY
See Also:
Constant Field Values
Constructor Detail

Fetcher

public Fetcher()

Fetcher

public Fetcher(Configuration conf)
Method Detail

configure

public void configure(JobConf job)
Specified by:
configure in interface JobConfigurable

close

public void close()

isParsing

public static boolean isParsing(Configuration conf)

isStoringContent

public static boolean isStoringContent(Configuration conf)

run

public void run(RecordReader input,
                OutputCollector output,
                Reporter reporter)
         throws IOException
Specified by:
run in interface MapRunnable
Throws:
IOException

fetch

public void fetch(Path segment,
                  int threads,
                  boolean parsing)
           throws IOException
Throws:
IOException

main

public static void main(String[] args)
                 throws Exception
Run the fetcher.

Throws:
Exception


Copyright © 2006 The Apache Software Foundation