# Uses of Class
org.apache.lucene.analysis.Tokenizer

Packages that use Tokenizer
Package
Description
org.apache.lucene.analysis
Text analysis.
org.apache.lucene.analysis.standard
Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

## Uses of Tokenizer in org.apache.lucene.analysis

Methods in org.apache.lucene.analysis that return Tokenizer
Modifier and Type
Method
Description
`final Tokenizer`
TokenizerFactory.`create()`
Creates a TokenStream of the specified input using the default attribute factory.
`abstract Tokenizer`
TokenizerFactory.`create(AttributeFactory factory)`
Creates a TokenStream of the specified input using the given AttributeFactory
Constructors in org.apache.lucene.analysis with parameters of type Tokenizer
Modifier
Constructor
Description
``
`TokenStreamComponents(Tokenizer tokenizer)`
Creates a new `Analyzer.TokenStreamComponents` from a Tokenizer
``
`TokenStreamComponents(Tokenizer tokenizer, TokenStream result)`
Creates a new `Analyzer.TokenStreamComponents` instance

## Uses of Tokenizer in org.apache.lucene.analysis.standard

Subclasses of Tokenizer in org.apache.lucene.analysis.standard
Modifier and Type
Class
Description
`final class`
`StandardTokenizer`
A grammar-based tokenizer constructed with JFlex.