Package org.apache.lucene.analysis.cn.smart

# Class HMMChineseTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.util.SegmentingTokenizerBase
org.apache.lucene.analysis.cn.smart.HMMChineseTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public class HMMChineseTokenizer extends SegmentingTokenizerBase
Tokenizer for Chinese or mixed Chinese-English text.

The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

## Nested Class Summary

## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

## Field Summary

### Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`buffer, BUFFERMAX, offset`

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

## Constructor Summary

Constructors
Constructor
Description
`HMMChineseTokenizer()`
Creates a new HMMChineseTokenizer
`HMMChineseTokenizer(AttributeFactory factory)`
Creates a new HMMChineseTokenizer, supplying the AttributeFactory

## Method Summary

Modifier and Type
Method
Description
`protected boolean`
`incrementWord()`
`void`
`reset()`
`protected void`
`setNextSentence(int sentenceStart, int sentenceEnd)`

### Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`end, incrementToken, isSafeEnd`

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`close, correctOffset, setReader, setReaderTestPoint`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

## Constructor Details

### HMMChineseTokenizer

public HMMChineseTokenizer()
Creates a new HMMChineseTokenizer

### HMMChineseTokenizer

public HMMChineseTokenizer(AttributeFactory factory)
Creates a new HMMChineseTokenizer, supplying the AttributeFactory

## Method Details

### setNextSentence

protected void setNextSentence(int sentenceStart, int sentenceEnd)

Specified by:
  `setNextSentence` in class `SegmentingTokenizerBase`

### incrementWord

protected boolean incrementWord()

Specified by:
  `incrementWord` in class `SegmentingTokenizerBase`

### reset

public void reset() throws IOException

Overrides:
  `reset` in class `SegmentingTokenizerBase`
Throws:
  `IOException`