Package org.apache.lucene.analysis.wikipedia

# Class WikipediaTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.wikipedia.WikipediaTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class WikipediaTokenizer extends Tokenizer
Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.

WARNING: This API is experimental and might change in incompatible ways in the next release.

## Nested Class Summary

## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

## Field Summary

Fields
Modifier and Type
Field
Description
`static final int`
`ACRONYM_ID`
`static final int`
`ALPHANUM_ID`
`static final int`
`APOSTROPHE_ID`
`static final String`
`BOLD`
`static final int`
`BOLD_ID`
`static final String`
`BOLD_ITALICS`
`static final int`
`BOLD_ITALICS_ID`
`static final int`
`BOTH`
Output the both the untokenized token and the splits
`static final String`
`CATEGORY`
`static final int`
`CATEGORY_ID`
`static final String`
`CITATION`
`static final int`
`CITATION_ID`
`static final int`
`CJ_ID`
`static final int`
`COMPANY_ID`
`static final int`
`EMAIL_ID`
`static final String`
`EXTERNAL_LINK`
`static final int`
`EXTERNAL_LINK_ID`
`static final String`
`EXTERNAL_LINK_URL`
`static final int`
`EXTERNAL_LINK_URL_ID`
`static final String`
`HEADING`
`static final int`
`HEADING_ID`
`static final int`
`HOST_ID`
`static final String`
`INTERNAL_LINK`
`static final int`
`INTERNAL_LINK_ID`
`static final String`
`ITALICS`
`static final int`
`ITALICS_ID`
`static final int`
`NUM_ID`
`static final String`
`SUB_HEADING`
`static final int`
`SUB_HEADING_ID`
`static final String[]`
`TOKEN_TYPES`
String token types that correspond to token type int constants
`static final int`
`TOKENS_ONLY`
Only output tokens
`static final int`
`UNTOKENIZED_ONLY`
Only output untokenized tokens, which are tokens that would normally be split into several tokens
`static final int`
`UNTOKENIZED_TOKEN_FLAG`
This flag is used to indicate that the produced "Token" would, if `TOKENS_ONLY` was used, produce multiple tokens.

### Fields inherited from class org.apache.lucene.analysis.Tokenizer

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

## Constructor Summary

Constructors
Constructor
Description
`WikipediaTokenizer()`
Creates a new instance of the `WikipediaTokenizer`.
`WikipediaTokenizer(int tokenOutput, Set<String> untokenizedTypes)`
Creates a new instance of the `WikipediaTokenizer`.
`WikipediaTokenizer(AttributeFactory factory, int tokenOutput, Set<String> untokenizedTypes)`
Creates a new instance of the `WikipediaTokenizer`.

## Method Summary

Modifier and Type
Method
Description
`void`
`close()`
`void`
`end()`
`final boolean`
`incrementToken()`
`void`
`reset()`

### Methods inherited from class org.apache.lucene.analysis.Tokenizer

`correctOffset, setReader, setReaderTestPoint`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

## Field Details

### INTERNAL_LINK

public static final String INTERNAL_LINK

See Also:
  - Constant Field Values

### EXTERNAL_LINK

public static final String EXTERNAL_LINK

See Also:
  - Constant Field Values

### EXTERNAL_LINK_URL

public static final String EXTERNAL_LINK_URL

See Also:
  - Constant Field Values

### CITATION

public static final String CITATION

See Also:
  - Constant Field Values

### CATEGORY

public static final String CATEGORY

See Also:
  - Constant Field Values

### BOLD

public static final String BOLD

See Also:
  - Constant Field Values

### ITALICS

public static final String ITALICS

See Also:
  - Constant Field Values

### BOLD_ITALICS

public static final String BOLD_ITALICS

See Also:
  - Constant Field Values

### HEADING

public static final String HEADING

See Also:
  - Constant Field Values

### SUB_HEADING

public static final String SUB_HEADING

See Also:
  - Constant Field Values

### ALPHANUM_ID

public static final int ALPHANUM_ID

See Also:
  - Constant Field Values

### APOSTROPHE_ID

public static final int APOSTROPHE_ID

See Also:
  - Constant Field Values

### ACRONYM_ID

public static final int ACRONYM_ID

See Also:
  - Constant Field Values

### COMPANY_ID

public static final int COMPANY_ID

See Also:
  - Constant Field Values

### EMAIL_ID

public static final int EMAIL_ID

See Also:
  - Constant Field Values

### HOST_ID

public static final int HOST_ID

See Also:
  - Constant Field Values

### NUM_ID

public static final int NUM_ID

See Also:
  - Constant Field Values

### CJ_ID

public static final int CJ_ID

See Also:
  - Constant Field Values

### INTERNAL_LINK_ID

public static final int INTERNAL_LINK_ID

See Also:
  - Constant Field Values

### EXTERNAL_LINK_ID

public static final int EXTERNAL_LINK_ID

See Also:
  - Constant Field Values

### CITATION_ID

public static final int CITATION_ID

See Also:
  - Constant Field Values

### CATEGORY_ID

public static final int CATEGORY_ID

See Also:
  - Constant Field Values

### BOLD_ID

public static final int BOLD_ID

See Also:
  - Constant Field Values

### ITALICS_ID

public static final int ITALICS_ID

See Also:
  - Constant Field Values

### BOLD_ITALICS_ID

public static final int BOLD_ITALICS_ID

See Also:
  - Constant Field Values

### HEADING_ID

public static final int HEADING_ID

See Also:
  - Constant Field Values

### SUB_HEADING_ID

public static final int SUB_HEADING_ID

See Also:
  - Constant Field Values

### EXTERNAL_LINK_URL_ID

public static final int EXTERNAL_LINK_URL_ID

See Also:
  - Constant Field Values

### TOKEN_TYPES

public static final String[] TOKEN_TYPES
String token types that correspond to token type int constants

### TOKENS_ONLY

public static final int TOKENS_ONLY
Only output tokens

See Also:
  - Constant Field Values

### UNTOKENIZED_ONLY

public static final int UNTOKENIZED_ONLY
Only output untokenized tokens, which are tokens that would normally be split into several tokens

See Also:
  - Constant Field Values

### BOTH

public static final int BOTH
Output the both the untokenized token and the splits

See Also:
  - Constant Field Values

### UNTOKENIZED_TOKEN_FLAG

public static final int UNTOKENIZED_TOKEN_FLAG
This flag is used to indicate that the produced "Token" would, if `TOKENS_ONLY` was used, produce multiple tokens.

See Also:
  - Constant Field Values

## Constructor Details

### WikipediaTokenizer

public WikipediaTokenizer()
Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to a newly created JFlex scanner.

### WikipediaTokenizer

public WikipediaTokenizer(int tokenOutput, Set<String> untokenizedTypes)
Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to the newly created JFlex scanner.

Parameters:
  `tokenOutput` - One of `TOKENS_ONLY`, `UNTOKENIZED_ONLY`, `BOTH`

### WikipediaTokenizer

public WikipediaTokenizer(AttributeFactory factory, int tokenOutput, Set<String> untokenizedTypes)
Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to the newly created JFlex scanner. Uses the given `AttributeFactory`.

Parameters:
  `tokenOutput` - One of `TOKENS_ONLY`, `UNTOKENIZED_ONLY`, `BOTH`

## Method Details

### incrementToken

public final boolean incrementToken() throws IOException

Specified by:
  `incrementToken` in class `TokenStream`
Throws:
  `IOException`

### close

public void close() throws IOException

Specified by:
  `close` in interface `AutoCloseable`
Specified by:
  `close` in interface `Closeable`
Overrides:
  `close` in class `Tokenizer`
Throws:
  `IOException`

### reset

public void reset() throws IOException

Overrides:
  `reset` in class `Tokenizer`
Throws:
  `IOException`

### end

public void end() throws IOException

Overrides:
  `end` in class `TokenStream`
Throws:
  `IOException`