Package org.apache.lucene.analysis

# Class TokenStreamToAutomaton

java.lang.Object
org.apache.lucene.analysis.TokenStreamToAutomaton

public class TokenStreamToAutomaton extends Object
Consumes a TokenStream and creates an `Automaton` where the transition labels are UTF8 bytes (or Unicode code points if unicodeArcs is true) from the `TermToBytesRefAttribute`. Between tokens we insert POS_SEP and for holes we insert HOLE.

WARNING: This API is experimental and might change in incompatible ways in the next release.

## Field Summary

Fields
Modifier and Type
Field
Description
`static final int`
`HOLE`
We add this arc to represent a hole.
`static final int`
`POS_SEP`
We create transition between two adjacent tokens.

## Constructor Summary

Constructors
Constructor
Description
`TokenStreamToAutomaton()`
Sole constructor.

## Method Summary

Modifier and Type
Method
Description
`protected BytesRef`
`changeToken(BytesRef in)`
Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.
`void`
`setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole)`
If true, any final offset gaps will result in adding a position hole.
`void`
`setPreservePositionIncrements(boolean enablePositionIncrements)`
Whether to generate holes in the automaton for missing positions, `true` by default.
`void`
`setUnicodeArcs(boolean unicodeArcs)`
Whether to make transition labels Unicode code points instead of UTF8 bytes, `false` by default
`Automaton`
`toAutomaton(TokenStream in)`
Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Field Details

### POS_SEP

public static final int POS_SEP
We create transition between two adjacent tokens.

See Also:
  - Constant Field Values

### HOLE

public static final int HOLE
We add this arc to represent a hole.

See Also:
  - Constant Field Values

## Constructor Details

### TokenStreamToAutomaton

public TokenStreamToAutomaton()
Sole constructor.

## Method Details

### setPreservePositionIncrements

public void setPreservePositionIncrements(boolean enablePositionIncrements)
Whether to generate holes in the automaton for missing positions, `true` by default.

### setFinalOffsetGapAsHole

public void setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole)
If true, any final offset gaps will result in adding a position hole.

### setUnicodeArcs

public void setUnicodeArcs(boolean unicodeArcs)
Whether to make transition labels Unicode code points instead of UTF8 bytes, `false` by default

### changeToken

protected BytesRef changeToken(BytesRef in)
Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.

### toAutomaton

public Automaton toAutomaton(TokenStream in) throws IOException
Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.

Throws:
  `IOException`