NAME

Mail::SpamAssassin::FuzzyHash::Util - Shared utilities for FuzzyHash algorithms

SYNOPSIS

use Mail::SpamAssassin::FuzzyHash::Util qw(normalize_tokens hamming_distance);

my @tokens = normalize_tokens($text);
my $dist   = hamming_distance($hex_a, $hex_b);

FUNCTIONS

normalize_tokens($text)

Normalises $text and returns a list of unique tokens suitable for fuzzy-hash computation.

Steps: ensure UTF-8 flag is set, apply NFKC normalisation (folds full-width ASCII spam chars and ligatures), lowercase, insert space boundaries around CJK characters, strip anything that is not a Unicode letter, digit, or space, collapse whitespace, split and deduplicate.

Returns ('__empty__') for texts that produce no tokens after normalisation.

hamming_distance($hex_a, $hex_b)

Returns the Hamming distance (number of differing bits) between two equal-length lowercase hex strings. The strings must have even length.