Package org.apache.lucene.analysis.th

# Class ThaiTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.util.SegmentingTokenizerBase
org.apache.lucene.analysis.th.ThaiTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public class ThaiTokenizer extends SegmentingTokenizerBase
Tokenizer that use `BreakIterator` to tokenize Thai text.

WARNING: this tokenizer may not be supported by all JREs. It is known to work with Sun/Oracle and Harmony JREs. If your application needs to be fully portable, consider using ICUTokenizer instead, which uses an ICU Thai BreakIterator that will always be available.

## Nested Class Summary

## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

## Field Summary

Fields
Modifier and Type
Field
Description
`static final boolean`
`DBBI_AVAILABLE`
True if the JRE supports a working dictionary-based breakiterator for Thai.

### Fields inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`buffer, BUFFERMAX, offset`

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

## Constructor Summary

Constructors
Constructor
Description
`ThaiTokenizer()`
Creates a new ThaiTokenizer
`ThaiTokenizer(AttributeFactory factory)`
Creates a new ThaiTokenizer, supplying the AttributeFactory

## Method Summary

Modifier and Type
Method
Description
`protected boolean`
`incrementWord()`
Returns true if another word is available
`protected void`
`setNextSentence(int sentenceStart, int sentenceEnd)`
Provides the next input sentence for analysis

### Methods inherited from class org.apache.lucene.analysis.util.SegmentingTokenizerBase

`end, incrementToken, isSafeEnd, reset`

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`close, correctOffset, setReader, setReaderTestPoint`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

## Field Details

### DBBI_AVAILABLE

public static final boolean DBBI_AVAILABLE
True if the JRE supports a working dictionary-based breakiterator for Thai. If this is false, this tokenizer will not work at all!

## Constructor Details

### ThaiTokenizer

public ThaiTokenizer()
Creates a new ThaiTokenizer

### ThaiTokenizer

public ThaiTokenizer(AttributeFactory factory)
Creates a new ThaiTokenizer, supplying the AttributeFactory

## Method Details

### setNextSentence

protected void setNextSentence(int sentenceStart, int sentenceEnd)
Description copied from class: `SegmentingTokenizerBase`
Provides the next input sentence for analysis

Specified by:
  `setNextSentence` in class `SegmentingTokenizerBase`

### incrementWord

protected boolean incrementWord()
Description copied from class: `SegmentingTokenizerBase`
Returns true if another word is available

Specified by:
  `incrementWord` in class `SegmentingTokenizerBase`