org.apache.lucene.search.similarities
Class BM25Similarity

java.lang.Object
  extended by org.apache.lucene.search.similarities.Similarity
      extended by org.apache.lucene.search.similarities.BM25Similarity

public class BM25Similarity
extends Similarity

BM25 Similarity. Introduced in Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC 1994). Gaithersburg, USA, November 1994.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.search.similarities.Similarity
Similarity.ExactSimScorer, Similarity.SimWeight, Similarity.SloppySimScorer
 
Field Summary
protected  boolean discountOverlaps
           
 
Constructor Summary
BM25Similarity()
          BM25 with these default values: k1 = 1.2, b = 0.75.
BM25Similarity(float k1, float b)
           
 
Method Summary
protected  float avgFieldLength(CollectionStatistics collectionStats)
          The default implementation computes the average as sumTotalTermFreq / maxDoc, or returns 1 if the index does not store sumTotalTermFreq (Lucene 3.x indexes or any field that omits frequency information).
 void computeNorm(FieldInvertState state, Norm norm)
          Computes the normalization value for a field, given the accumulated state of term processing for this field (see FieldInvertState).
 Similarity.SimWeight computeWeight(float queryBoost, CollectionStatistics collectionStats, TermStatistics... termStats)
          Compute any collection-level weight (e.g.
protected  float decodeNormValue(byte b)
          The default implementation returns 1 / f2 where f is SmallFloat.byte315ToFloat(byte).
protected  byte encodeNormValue(float boost, int fieldLength)
          The default implementation encodes boost / sqrt(length) with SmallFloat.floatToByte315(float).
 Similarity.ExactSimScorer exactSimScorer(Similarity.SimWeight stats, AtomicReaderContext context)
          Creates a new Similarity.ExactSimScorer to score matching documents from a segment of the inverted index.
 float getB()
           
 boolean getDiscountOverlaps()
           
 float getK1()
           
protected  float idf(long docFreq, long numDocs)
          Implemented as log(1 + (numDocs - docFreq + 0.5)/(docFreq + 0.5)).
 Explanation idfExplain(CollectionStatistics collectionStats, TermStatistics termStats)
           
 Explanation idfExplain(CollectionStatistics collectionStats, TermStatistics[] termStats)
           
protected  float scorePayload(int doc, int start, int end, BytesRef payload)
          The default implementation returns 1
 void setDiscountOverlaps(boolean v)
          Determines whether overlap tokens (Tokens with 0 position increment) are ignored when computing norm.
protected  float sloppyFreq(int distance)
          Implemented as 1 / (distance + 1).
 Similarity.SloppySimScorer sloppySimScorer(Similarity.SimWeight stats, AtomicReaderContext context)
          Creates a new Similarity.SloppySimScorer to score matching documents from a segment of the inverted index.
 String toString()
           
 
Methods inherited from class org.apache.lucene.search.similarities.Similarity
coord, queryNorm
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

discountOverlaps

protected boolean discountOverlaps
Constructor Detail

BM25Similarity

public BM25Similarity(float k1,
                      float b)

BM25Similarity

public BM25Similarity()
BM25 with these default values:

Method Detail

idf

protected float idf(long docFreq,
                    long numDocs)
Implemented as log(1 + (numDocs - docFreq + 0.5)/(docFreq + 0.5)).


sloppyFreq

protected float sloppyFreq(int distance)
Implemented as 1 / (distance + 1).


scorePayload

protected float scorePayload(int doc,
                             int start,
                             int end,
                             BytesRef payload)
The default implementation returns 1


avgFieldLength

protected float avgFieldLength(CollectionStatistics collectionStats)
The default implementation computes the average as sumTotalTermFreq / maxDoc, or returns 1 if the index does not store sumTotalTermFreq (Lucene 3.x indexes or any field that omits frequency information).


encodeNormValue

protected byte encodeNormValue(float boost,
                               int fieldLength)
The default implementation encodes boost / sqrt(length) with SmallFloat.floatToByte315(float). This is compatible with Lucene's default implementation. If you change this, then you should change decodeNormValue(byte) to match.


decodeNormValue

protected float decodeNormValue(byte b)
The default implementation returns 1 / f2 where f is SmallFloat.byte315ToFloat(byte).


setDiscountOverlaps

public void setDiscountOverlaps(boolean v)
Determines whether overlap tokens (Tokens with 0 position increment) are ignored when computing norm. By default this is true, meaning overlap tokens do not count when computing norms.


getDiscountOverlaps

public boolean getDiscountOverlaps()
See Also:
setDiscountOverlaps(boolean)

computeNorm

public final void computeNorm(FieldInvertState state,
                              Norm norm)
Description copied from class: Similarity
Computes the normalization value for a field, given the accumulated state of term processing for this field (see FieldInvertState).

Implementations should calculate a norm value based on the field state and set that value to the given Norm.

Matches in longer fields are less precise, so implementations of this method usually set smaller values when state.getLength() is large, and larger values when state.getLength() is small.

Specified by:
computeNorm in class Similarity
Parameters:
state - current processing state for this field
norm - holds the computed norm value when this method returns

idfExplain

public Explanation idfExplain(CollectionStatistics collectionStats,
                              TermStatistics termStats)

idfExplain

public Explanation idfExplain(CollectionStatistics collectionStats,
                              TermStatistics[] termStats)

computeWeight

public final Similarity.SimWeight computeWeight(float queryBoost,
                                                CollectionStatistics collectionStats,
                                                TermStatistics... termStats)
Description copied from class: Similarity
Compute any collection-level weight (e.g. IDF, average document length, etc) needed for scoring a query.

Specified by:
computeWeight in class Similarity
Parameters:
queryBoost - the query-time boost.
collectionStats - collection-level statistics, such as the number of tokens in the collection.
termStats - term-level statistics, such as the document frequency of a term across the collection.
Returns:
SimWeight object with the information this Similarity needs to score a query.

exactSimScorer

public final Similarity.ExactSimScorer exactSimScorer(Similarity.SimWeight stats,
                                                      AtomicReaderContext context)
                                               throws IOException
Description copied from class: Similarity
Creates a new Similarity.ExactSimScorer to score matching documents from a segment of the inverted index.

Specified by:
exactSimScorer in class Similarity
Parameters:
stats - collection information from Similarity.computeWeight(float, CollectionStatistics, TermStatistics...)
context - segment of the inverted index to be scored.
Returns:
ExactSimScorer for scoring documents across context
Throws:
IOException

sloppySimScorer

public final Similarity.SloppySimScorer sloppySimScorer(Similarity.SimWeight stats,
                                                        AtomicReaderContext context)
                                                 throws IOException
Description copied from class: Similarity
Creates a new Similarity.SloppySimScorer to score matching documents from a segment of the inverted index.

Specified by:
sloppySimScorer in class Similarity
Parameters:
stats - collection information from Similarity.computeWeight(float, CollectionStatistics, TermStatistics...)
context - segment of the inverted index to be scored.
Returns:
SloppySimScorer for scoring documents across context
Throws:
IOException

toString

public String toString()
Overrides:
toString in class Object

getK1

public float getK1()

getB

public float getB()


Copyright © 2000-2012 Apache Software Foundation. All Rights Reserved.