org.apache.lucene.analysis
Class Analyzer

java.lang.Object
  extended by org.apache.lucene.analysis.Analyzer
Direct Known Subclasses:
AnalyzerWrapper

public abstract class Analyzer
extends Object

An Analyzer builds TokenStreams, which analyze text. It thus represents a policy for extracting index terms from text.

In order to define what analysis is done, subclasses must define their Analyzer.TokenStreamComponents in createComponents(String, Reader). The components are then reused in each call to tokenStream(String, Reader).


Nested Class Summary
static class Analyzer.GlobalReuseStrategy
          Implementation of Analyzer.ReuseStrategy that reuses the same components for every field.
static class Analyzer.PerFieldReuseStrategy
          Implementation of Analyzer.ReuseStrategy that reuses components per-field by maintaining a Map of TokenStreamComponent per field name.
static class Analyzer.ReuseStrategy
          Strategy defining how TokenStreamComponents are reused per call to tokenStream(String, java.io.Reader).
static class Analyzer.TokenStreamComponents
          This class encapsulates the outer components of a token stream.
 
Constructor Summary
Analyzer()
           
Analyzer(Analyzer.ReuseStrategy reuseStrategy)
           
 
Method Summary
 void close()
          Frees persistent resources used by this Analyzer
protected abstract  Analyzer.TokenStreamComponents createComponents(String fieldName, Reader reader)
          Creates a new Analyzer.TokenStreamComponents instance for this analyzer.
 int getOffsetGap(String fieldName)
          Just like getPositionIncrementGap(java.lang.String), except for Token offsets instead.
 int getPositionIncrementGap(String fieldName)
          Invoked before indexing a IndexableField instance if terms have already been added to that field.
protected  Reader initReader(String fieldName, Reader reader)
          Override this if you want to add a CharFilter chain.
 TokenStream tokenStream(String fieldName, Reader reader)
          Creates a TokenStream that is allowed to be re-use from the previous time that the same thread called this method.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Analyzer

public Analyzer()

Analyzer

public Analyzer(Analyzer.ReuseStrategy reuseStrategy)
Method Detail

createComponents

protected abstract Analyzer.TokenStreamComponents createComponents(String fieldName,
                                                                   Reader reader)
Creates a new Analyzer.TokenStreamComponents instance for this analyzer.

Parameters:
fieldName - the name of the fields content passed to the Analyzer.TokenStreamComponents sink as a reader
reader - the reader passed to the Tokenizer constructor
Returns:
the Analyzer.TokenStreamComponents for this analyzer.

tokenStream

public final TokenStream tokenStream(String fieldName,
                                     Reader reader)
                              throws IOException
Creates a TokenStream that is allowed to be re-use from the previous time that the same thread called this method. Callers that do not need to use more than one TokenStream at the same time from this analyzer should use this method for better performance.

This method uses createComponents(String, Reader) to obtain an instance of Analyzer.TokenStreamComponents. It returns the sink of the components and stores the components internally. Subsequent calls to this method will reuse the previously stored components after resetting them through Analyzer.TokenStreamComponents.setReader(Reader).

Parameters:
fieldName - the name of the field the created TokenStream is used for
reader - the reader the streams source reads from
Throws:
IOException

initReader

protected Reader initReader(String fieldName,
                            Reader reader)
Override this if you want to add a CharFilter chain.


getPositionIncrementGap

public int getPositionIncrementGap(String fieldName)
Invoked before indexing a IndexableField instance if terms have already been added to that field. This allows custom analyzers to place an automatic position increment gap between IndexbleField instances using the same field name. The default value position increment gap is 0. With a 0 position increment gap and the typical default token position increment of 1, all terms in a field, including across IndexableField instances, are in successive positions, allowing exact PhraseQuery matches, for instance, across IndexableField instance boundaries.

Parameters:
fieldName - IndexableField name being indexed.
Returns:
position increment gap, added to the next token emitted from tokenStream(String,Reader)

getOffsetGap

public int getOffsetGap(String fieldName)
Just like getPositionIncrementGap(java.lang.String), except for Token offsets instead. By default this returns 1. This method is only called if the field produced at least one token for indexing.

Parameters:
fieldName - the field just indexed
Returns:
offset gap, added to the next token emitted from tokenStream(String,Reader)

close

public void close()
Frees persistent resources used by this Analyzer



Copyright © 2000-2012 Apache Software Foundation. All Rights Reserved.