NAME

FuzzyHash::ZOrder - Fuzzy text similarity via DNS, using exactly 4 queries

SYNOPSIS

use FuzzyHash::ZOrder qw(zorder_digest zorder_compare zorder_dns_keys);

my $query_digest = zorder_digest($incoming_text);
for my $key (zorder_dns_keys($incoming_text, 'fbl.example.com')) {
  for my $stored_digest (dns_txt_lookup($key)) {
    my $sim = zorder_compare($query_digest, $stored_digest);
    if ($sim >= 85) { # rule fires }
  }
}

DESCRIPTION

FuzzyHash::ZOrder detects texts with ~90% token overlap using exactly 4 DNS TXT lookups followed by a local similarity check via zorder_compare().

Algorithm

Normalisation

Lowercase, replace non-alphanumeric with spaces, collapse whitespace, deduplicate tokens.

32-bit MinHash signature

32 independent hash functions are applied to the deduplicated token set. Each function finds the token with the minimum hash value; the LSB parity of that minimum becomes one output bit. The result is a 32-bit signature. Each bit agrees between two texts with probability exactly equal to their Jaccard similarity.

4 bands x 8 bits

The 32-bit signature is split into 4 consecutive 8-bit bands. Two texts share a band key when all 8 bits in that band agree: probability S^8 at Jaccard similarity S.

Local similarity verification

Each TXT response carries the full 32-bit digest of the indexed string. zorder_compare() verifies similarity locally using all 32 bits. Unrelated texts will have a score of ~50%.

FUNCTIONS

zorder_digest($text)

Computes the 32-bit MinHash signature of $text. Returns an 8-character lowercase hex string.

This is the value stored in each DNS TXT record at index time and used for local similarity verification at query time.

zorder_dns_keys($text, $domain)

Returns a list of exactly 4 DNS FQDNs for $text under $domain.

Use for both indexing (store digest as TXT at each key) and querying (look up each key, verify TXT responses locally with zorder_compare).

zorder_body_band_labels($text)

Returns a list of exactly 4 bare band labels without a trailing domain, each of the form fz2BVV.

zorder_compare($digest_a, $digest_b)

Compares two 8-char hex digests. Returns integer similarity 0-100, computed as 100 * (1 - hamming_distance / 32).

100  = identical token sets (0 bits differ)
 91  = ~3 bits differ  -> very high overlap
 85  = ~5 bits differ  -> high overlap
 75  = ~8 bits differ  -> moderate overlap
 50  = ~16 bits differ -> noise floor for unrelated texts

The noise floor at ~50% provides a clean gap below the suggested 90% threshold, so coincidental DNS band hits from unrelated indexed strings are reliably rejected without any additional DNS queries.

zorder_explain($digest_a, $digest_b)

Returns a human-readable breakdown: Hamming distance, similarity score, shared band count, firing decision, and a per-band bit-agreement table. Used only for debugging RBL entries and tuning the similarity threshold.