# Uses of Class
org.apache.lucene.analysis.TokenStream

Packages that use TokenStream
Package
Description
org.apache.lucene.analysis
Text analysis.
org.apache.lucene.analysis.standard
Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.
org.apache.lucene.document
The logical representation of a `Document` for indexing and searching.
org.apache.lucene.index
Code to maintain and access indices.
org.apache.lucene.util
Some utility classes.
org.apache.lucene.util.graph
Utility classes for working with token streams as graphs.

## Uses of TokenStream in org.apache.lucene.analysis

Subclasses of TokenStream in org.apache.lucene.analysis
Modifier and Type
Class
Description
`final class`
`CachingTokenFilter`
This class can be used if the token attributes of a TokenStream are intended to be consumed more than once.
`class`
`FilteringTokenFilter`
Abstract base class for TokenFilters that may remove tokens.
`class`
`GraphTokenFilter`
An abstract TokenFilter that exposes its input stream as a graph
`class`
`LowerCaseFilter`
Normalizes token text to lower case.
`class`
`StopFilter`
Removes stop words from a token stream.
`class`
`TokenFilter`
A TokenFilter is a TokenStream whose input is another TokenStream.
`class`
`Tokenizer`
A Tokenizer is a TokenStream whose input is a Reader.
Fields in org.apache.lucene.analysis declared as TokenStream
Modifier and Type
Field
Description
`protected final TokenStream`
TokenFilter.`input`
The source of tokens for this filter.
`protected final TokenStream`
Analyzer.TokenStreamComponents.`sink`
Sink tokenstream, such as the outer tokenfilter decorating the chain.
Methods in org.apache.lucene.analysis that return TokenStream
Modifier and Type
Method
Description
`abstract TokenStream`
TokenFilterFactory.`create(TokenStream input)`
Transform the specified input TokenStream
`TokenStream`
Analyzer.TokenStreamComponents.`getTokenStream()`
Returns the sink `TokenStream`
`protected TokenStream`
Analyzer.`normalize(String fieldName, TokenStream in)`
Wrap the given `TokenStream` in order to apply normalization filters.
`protected final TokenStream`
AnalyzerWrapper.`normalize(String fieldName, TokenStream in)`
`TokenStream`
TokenFilterFactory.`normalize(TokenStream input)`
Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to `create` method.
`final TokenStream`
Analyzer.`tokenStream(String fieldName, Reader reader)`
Returns a TokenStream suitable for `fieldName`, tokenizing the contents of `reader`.
`final TokenStream`
Analyzer.`tokenStream(String fieldName, String text)`
Returns a TokenStream suitable for `fieldName`, tokenizing the contents of `text`.
`static TokenStream`
AutomatonToTokenStream.`toTokenStream(Automaton automaton)`
converts an automaton into a TokenStream.
`TokenStream`
TokenFilter.`unwrap()`
`protected TokenStream`
AnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.
`protected final TokenStream`
DelegatingAnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
Methods in org.apache.lucene.analysis with parameters of type TokenStream
Modifier and Type
Method
Description
`abstract TokenStream`
TokenFilterFactory.`create(TokenStream input)`
Transform the specified input TokenStream
`protected TokenStream`
Analyzer.`normalize(String fieldName, TokenStream in)`
Wrap the given `TokenStream` in order to apply normalization filters.
`protected final TokenStream`
AnalyzerWrapper.`normalize(String fieldName, TokenStream in)`
`TokenStream`
TokenFilterFactory.`normalize(TokenStream input)`
Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to `create` method.
`Automaton`
TokenStreamToAutomaton.`toAutomaton(TokenStream in)`
Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.
`protected TokenStream`
AnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.
`protected final TokenStream`
DelegatingAnalyzerWrapper.`wrapTokenStreamForNormalization(String fieldName, TokenStream in)`
Constructors in org.apache.lucene.analysis with parameters of type TokenStream
Modifier
Constructor
Description
``
`CachingTokenFilter(TokenStream input)`
Create a new CachingTokenFilter around `input`.
``
`FilteringTokenFilter(TokenStream in)`
Create a new `FilteringTokenFilter`.
``
`GraphTokenFilter(TokenStream input)`
Create a new GraphTokenFilter
``
`LowerCaseFilter(TokenStream in)`
Create a new LowerCaseFilter, that normalizes token text to lower case.
``
`StopFilter(TokenStream in, CharArraySet stopWords)`
Constructs a filter which removes words from the input TokenStream that are named in the Set.
`protected`
`TokenFilter(TokenStream input)`
Construct a token stream filtering the given input.
``
`TokenStreamComponents(Consumer<Reader> source, TokenStream result)`
Creates a new `Analyzer.TokenStreamComponents` instance.
``
`TokenStreamComponents(Tokenizer tokenizer, TokenStream result)`
Creates a new `Analyzer.TokenStreamComponents` instance

## Uses of TokenStream in org.apache.lucene.analysis.standard

Subclasses of TokenStream in org.apache.lucene.analysis.standard
Modifier and Type
Class
Description
`final class`
`StandardTokenizer`
A grammar-based tokenizer constructed with JFlex.
Methods in org.apache.lucene.analysis.standard that return TokenStream
Modifier and Type
Method
Description
`protected TokenStream`
StandardAnalyzer.`normalize(String fieldName, TokenStream in)`
Methods in org.apache.lucene.analysis.standard with parameters of type TokenStream
Modifier and Type
Method
Description
`protected TokenStream`
StandardAnalyzer.`normalize(String fieldName, TokenStream in)`

## Uses of TokenStream in org.apache.lucene.document

Methods in org.apache.lucene.document that return TokenStream
Modifier and Type
Method
Description
`TokenStream`
FeatureField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
`TokenStream`
Field.`tokenStream(Analyzer analyzer, TokenStream reuse)`
`TokenStream`
ShapeDocValuesField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
TokenStreams are not yet supported
`TokenStream`
Field.`tokenStreamValue()`
The TokenStream for this field to be used when indexing, or null.
Methods in org.apache.lucene.document with parameters of type TokenStream
Modifier and Type
Method
Description
`void`
Field.`setTokenStream(TokenStream tokenStream)`
Expert: sets the token stream to be used for indexing.
`TokenStream`
FeatureField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
`TokenStream`
Field.`tokenStream(Analyzer analyzer, TokenStream reuse)`
`TokenStream`
ShapeDocValuesField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
TokenStreams are not yet supported
Constructors in org.apache.lucene.document with parameters of type TokenStream
Modifier
Constructor
Description
``
`Field(String name, TokenStream tokenStream, IndexableFieldType type)`
Create field with TokenStream value.
``
`TextField(String name, TokenStream stream)`
Creates a new un-stored TextField with TokenStream value.

## Uses of TokenStream in org.apache.lucene.index

Methods in org.apache.lucene.index that return TokenStream
Modifier and Type
Method
Description
`TokenStream`
IndexableField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
Creates the TokenStream used for indexing this field.
Methods in org.apache.lucene.index with parameters of type TokenStream
Modifier and Type
Method
Description
`TokenStream`
IndexableField.`tokenStream(Analyzer analyzer, TokenStream reuse)`
Creates the TokenStream used for indexing this field.

## Uses of TokenStream in org.apache.lucene.util

Methods in org.apache.lucene.util with parameters of type TokenStream
Modifier and Type
Method
Description
`protected Query`
QueryBuilder.`analyzeBoolean(String field, TokenStream stream)`
Creates simple boolean query from the cached tokenstream contents
`protected Query`
QueryBuilder.`analyzeGraphBoolean(String field, TokenStream source, BooleanClause.Occur operator)`
Creates a boolean query from a graph token stream.
`protected Query`
QueryBuilder.`analyzeGraphPhrase(TokenStream source, String field, int phraseSlop)`
Creates graph phrase query from the tokenstream contents
`protected Query`
QueryBuilder.`analyzeMultiBoolean(String field, TokenStream stream, BooleanClause.Occur operator)`
Creates complex boolean query from the cached tokenstream contents
`protected Query`
QueryBuilder.`analyzeMultiPhrase(String field, TokenStream stream, int slop)`
Creates complex phrase query from the cached tokenstream contents
`protected Query`
QueryBuilder.`analyzePhrase(String field, TokenStream stream, int slop)`
Creates simple phrase query from the cached tokenstream contents
`protected Query`
QueryBuilder.`analyzeTerm(String field, TokenStream stream)`
Creates simple term query from the cached tokenstream contents
`protected Query`
QueryBuilder.`createFieldQuery(TokenStream source, BooleanClause.Occur operator, String field, boolean quoted, int phraseSlop)`
Creates a query from a token stream.

## Uses of TokenStream in org.apache.lucene.util.graph

Methods in org.apache.lucene.util.graph that return types with arguments of type TokenStream
Modifier and Type
Method
Description
`Iterator<TokenStream>`
GraphTokenStreamFiniteStrings.`getFiniteStrings()`
Get all finite strings from the automaton.
`Iterator<TokenStream>`
GraphTokenStreamFiniteStrings.`getFiniteStrings(int startState, int endState)`
Get all finite strings that start at `startState` and end at `endState`.
Constructors in org.apache.lucene.util.graph with parameters of type TokenStream
Modifier
Constructor
Description
``
`GraphTokenStreamFiniteStrings(TokenStream in)`