Uses of Interface
opennlp.tools.tokenize.Tokenizer
-
Packages that use Tokenizer Package Description opennlp.tools.cmdline.parser opennlp.tools.formats.brat opennlp.tools.formats.muc opennlp.tools.tokenize Contains classes related to finding token or words in a string.opennlp.tools.util.featuregen This package contains classes for generating sequence features. -
-
Uses of Tokenizer in opennlp.tools.cmdline.parser
Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer Modifier and Type Method Description static Parse[]
ParserTool. parseLine(String line, Parser parser, Tokenizer tokenizer, int numParses)
-
Uses of Tokenizer in opennlp.tools.formats.brat
Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer Constructor Description BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer)
BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)
BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)
Creates a newBratNameSampleStream
.BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)
Creates a newBratNameSampleStream
. -
Uses of Tokenizer in opennlp.tools.formats.muc
Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer Constructor Description MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)
MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)
-
Uses of Tokenizer in opennlp.tools.tokenize
Classes in opennlp.tools.tokenize that implement Tokenizer Modifier and Type Class Description class
SimpleTokenizer
Performs tokenization using character classes.class
TokenizerME
A Tokenizer for converting raw text into separated tokens.class
WhitespaceTokenizer
This tokenizer uses white spaces to tokenize the input text.class
WordpieceTokenizer
A WordPiece tokenizer.Constructors in opennlp.tools.tokenize with parameters of type Tokenizer Constructor Description TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)
Initializes the current instance with the givenTokenizer
.TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)
-
Uses of Tokenizer in opennlp.tools.util.featuregen
Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer Constructor Description TokenPatternFeatureGenerator(Tokenizer supportTokenizer)
Initializes a new instance.
-