Package org.apache.lucene.analysis.ja

# Class JapaneseTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.ja.JapaneseTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class JapaneseTokenizer extends Tokenizer
Tokenizer for Japanese that uses morphological analysis.

This tokenizer sets a number of additional attributes:

- `BaseFormAttribute` containing base form for inflected adjectives and verbs.
- `PartOfSpeechAttribute` containing part-of-speech.
- `ReadingAttribute` containing reading and pronunciation.
- `InflectionAttribute` containing additional part-of-speech information for inflected forms.

This tokenizer uses a rolling Viterbi search to find the least cost segmentation (path) of the incoming characters. For tokens that appear to be compound (> length 2 for all Kanji, or > length 7 for non-Kanji), we see if there is a 2nd best segmentation of that token after applying penalties to the long tokens. If so, and the Mode is `JapaneseTokenizer.Mode.SEARCH`, we output the alternate segmentation as well.

## Nested Class Summary

Nested Classes
Modifier and Type
Class
Description
`static enum`
`JapaneseTokenizer.Mode`
Tokenization mode: this determines how the tokenizer handles compound and unknown words.

## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

## Field Summary

Fields
Modifier and Type
Field
Description
`static final JapaneseTokenizer.Mode`
`DEFAULT_MODE`
Default tokenization mode.

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

## Constructor Summary

Constructors
Constructor
Description
`JapaneseTokenizer(UserDictionary userDictionary, boolean discardPunctuation, boolean discardCompoundToken, JapaneseTokenizer.Mode mode)`
Create a new JapaneseTokenizer.
`JapaneseTokenizer(UserDictionary userDictionary, boolean discardPunctuation, JapaneseTokenizer.Mode mode)`
Create a new JapaneseTokenizer.
`JapaneseTokenizer(AttributeFactory factory, TokenInfoDictionary systemDictionary, UnknownDictionary unkDictionary, ConnectionCosts connectionCosts, UserDictionary userDictionary, boolean discardPunctuation, boolean discardCompoundToken, JapaneseTokenizer.Mode mode)`
Create a new JapaneseTokenizer, supplying a custom system dictionary and unknown dictionary.
`JapaneseTokenizer(AttributeFactory factory, UserDictionary userDictionary, boolean discardPunctuation, boolean discardCompoundToken, JapaneseTokenizer.Mode mode)`
Create a new JapaneseTokenizer using the system and unknown dictionaries shipped with Lucene.
`JapaneseTokenizer(AttributeFactory factory, UserDictionary userDictionary, boolean discardPunctuation, JapaneseTokenizer.Mode mode)`
Create a new JapaneseTokenizer using the system and unknown dictionaries shipped with Lucene.

## Method Summary

Modifier and Type
Method
Description
`int`
`calcNBestCost(String examples)`
`void`
`close()`
`void`
`end()`
`boolean`
`incrementToken()`
`void`
`reset()`
`void`
`setGraphvizFormatter(GraphvizFormatter<JaMorphData> dotOut)`
Expert: set this to produce graphviz (dot) output of the Viterbi lattice
`void`
`setNBestCost(int value)`

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`correctOffset, setReader, setReaderTestPoint`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

## Field Details

### DEFAULT_MODE

public static final JapaneseTokenizer.Mode DEFAULT_MODE
Default tokenization mode. Currently this is `JapaneseTokenizer.Mode.SEARCH`.

## Constructor Details

### JapaneseTokenizer

public JapaneseTokenizer(UserDictionary userDictionary, boolean discardPunctuation, JapaneseTokenizer.Mode mode)
Create a new JapaneseTokenizer.

Uses the default AttributeFactory.

Parameters:
  `userDictionary` - Optional: if non-null, user dictionary.
  `discardPunctuation` - true if punctuation tokens should be dropped from the output.
  `mode` - tokenization mode.

### JapaneseTokenizer

public JapaneseTokenizer(UserDictionary userDictionary, boolean discardPunctuation, boolean discardCompoundToken, JapaneseTokenizer.Mode mode)
Create a new JapaneseTokenizer.

Uses the default AttributeFactory.

Parameters:
  `userDictionary` - Optional: if non-null, user dictionary.
  `discardPunctuation` - true if punctuation tokens should be dropped from the output.
  `discardCompoundToken` - true if compound tokens should be dropped from the output when tokenization mode is not NORMAL.
  `mode` - tokenization mode.

### JapaneseTokenizer

public JapaneseTokenizer(AttributeFactory factory, UserDictionary userDictionary, boolean discardPunctuation, JapaneseTokenizer.Mode mode)
Create a new JapaneseTokenizer using the system and unknown dictionaries shipped with Lucene.

Parameters:
  `factory` - the AttributeFactory to use
  `userDictionary` - Optional: if non-null, user dictionary.
  `discardPunctuation` - true if punctuation tokens should be dropped from the output.
  `mode` - tokenization mode.

### JapaneseTokenizer

public JapaneseTokenizer(AttributeFactory factory, UserDictionary userDictionary, boolean discardPunctuation, boolean discardCompoundToken, JapaneseTokenizer.Mode mode)
Create a new JapaneseTokenizer using the system and unknown dictionaries shipped with Lucene.

Parameters:
  `factory` - the AttributeFactory to use
  `userDictionary` - Optional: if non-null, user dictionary.
  `discardPunctuation` - true if punctuation tokens should be dropped from the output.
  `discardCompoundToken` - true if compound tokens should be dropped from the output when tokenization mode is not NORMAL.
  `mode` - tokenization mode.

### JapaneseTokenizer

public JapaneseTokenizer(AttributeFactory factory, TokenInfoDictionary systemDictionary, UnknownDictionary unkDictionary, ConnectionCosts connectionCosts, UserDictionary userDictionary, boolean discardPunctuation, boolean discardCompoundToken, JapaneseTokenizer.Mode mode)
Create a new JapaneseTokenizer, supplying a custom system dictionary and unknown dictionary. This constructor provides an entry point for users that want to construct custom language models that can be used as input to `DictionaryBuilder`.

Parameters:
  `factory` - the AttributeFactory to use
  `systemDictionary` - a custom known token dictionary
  `unkDictionary` - a custom unknown token dictionary
  `connectionCosts` - custom token transition costs
  `userDictionary` - Optional: if non-null, user dictionary.
  `discardPunctuation` - true if punctuation tokens should be dropped from the output.
  `discardCompoundToken` - true if compound tokens should be dropped from the output when tokenization mode is not NORMAL.
  `mode` - tokenization mode.
WARNING: This API is experimental and might change in incompatible ways in the next release.

## Method Details

### setGraphvizFormatter

public void setGraphvizFormatter(GraphvizFormatter<JaMorphData> dotOut)
Expert: set this to produce graphviz (dot) output of the Viterbi lattice

### close

public void close() throws IOException

Specified by:
  `close` in interface `AutoCloseable`
Specified by:
  `close` in interface `Closeable`
Overrides:
  `close` in class `Tokenizer`
Throws:
  `IOException`

### reset

public void reset() throws IOException

Overrides:
  `reset` in class `Tokenizer`
Throws:
  `IOException`

### end

public void end() throws IOException

Overrides:
  `end` in class `TokenStream`
Throws:
  `IOException`

### incrementToken

public boolean incrementToken() throws IOException

Specified by:
  `incrementToken` in class `TokenStream`
Throws:
  `IOException`

### calcNBestCost

public int calcNBestCost(String examples)

### setNBestCost

public void setNBestCost(int value)