Package org.apache.lucene.analysis.minhash

# Class MinHashFilter

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.TokenFilter
org.apache.lucene.analysis.minhash.MinHashFilter

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`, `Unwrappable<TokenStream>`

public class MinHashFilter extends TokenFilter
Generate min hash tokens from an incoming stream of tokens. The incoming tokens would typically be 5 word shingles.

The number of hashes used and the number of minimum values for each hash can be set. You could have 1 hash and keep the 100 lowest values or 100 hashes and keep the lowest one for each. Hashes can also be bucketed in ranges over the 128-bit hash space,

A 128-bit hash is used internally. 5 word shingles from 10e5 words generate 10e25 combinations So a 64 bit hash would have collisions (1.8e19)

When using different hashes 32 bits are used for the hash position leaving scope for 8e28 unique hashes. A single hash will use all 128 bits.

## Nested Class Summary

## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

`AttributeSource.State`

## Field Summary

Fields
Modifier and Type
Field
Description
`static final int`
`DEFAULT_BUCKET_COUNT`
`static final int`
`DEFAULT_HASH_COUNT`
`static final int`
`DEFAULT_HASH_SET_SIZE`

### Fields inherited from class org.apache.lucene.analysis.TokenFilter

`input`

### Fields inherited from class org.apache.lucene.analysis.TokenStream

`DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

## Constructor Summary

Constructors
Constructor
Description
`MinHashFilter(TokenStream input, int hashCount, int bucketCount, int hashSetSize, boolean withRotation)`
create a MinHash filter

## Method Summary

Modifier and Type
Method
Description
`void`
`end()`
`final boolean`
`incrementToken()`
`void`
`reset()`

### Methods inherited from class org.apache.lucene.analysis.TokenFilter

`close, unwrap`

### Methods inherited from class org.apache.lucene.util.AttributeSource

`addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

### Methods inherited from class java.lang.Object

`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

## Field Details

### DEFAULT_HASH_COUNT

public static final int DEFAULT_HASH_COUNT

See Also:
  - Constant Field Values

### DEFAULT_HASH_SET_SIZE

public static final int DEFAULT_HASH_SET_SIZE

See Also:
  - Constant Field Values

### DEFAULT_BUCKET_COUNT

public static final int DEFAULT_BUCKET_COUNT

See Also:
  - Constant Field Values

## Constructor Details

### MinHashFilter

public MinHashFilter(TokenStream input, int hashCount, int bucketCount, int hashSetSize, boolean withRotation)
create a MinHash filter

Parameters:
  `input` - the token stream
  `hashCount` - the no. of hashes
  `bucketCount` - the no. of buckets for hashing
  `hashSetSize` - the no. of min hashes to keep
  `withRotation` - whether rotate or not hashes while incrementing tokens

## Method Details

### incrementToken

public final boolean incrementToken() throws IOException

Specified by:
  `incrementToken` in class `TokenStream`
Throws:
  `IOException`

### end

public void end() throws IOException

Overrides:
  `end` in class `TokenFilter`
Throws:
  `IOException`

### reset

public void reset() throws IOException

Overrides:
  `reset` in class `TokenFilter`
Throws:
  `IOException`