Package org.apache.lucene.analysis.cn.smart.hhmm

# Class SegTokenFilter

java.lang.Object
org.apache.lucene.analysis.cn.smart.hhmm.SegTokenFilter

public class SegTokenFilter extends Object
Filters a `SegToken` by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into `Utility.COMMON_DELIMITER`

WARNING: This API is experimental and might change in incompatible ways in the next release.

## Constructor Summary

Constructors
Constructor
Description
`SegTokenFilter()`

## Method Summary

Modifier and Type
Method
Description
`SegToken`
`filter(SegToken token)`
Filter an input `SegToken`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Constructor Details

### SegTokenFilter

public SegTokenFilter()

## Method Details

### filter

public SegToken filter(SegToken token)
Filter an input `SegToken`

Full-width latin will be converted to half-width, then all latin will be lowercased. All punctuation is converted into `Utility.COMMON_DELIMITER`

Parameters:
  `token` - input `SegToken`
Returns:
  normalized `SegToken`