NAME
Algorithm::ToNumberMunger - Compile declarative specs into closures that munge raw values into numbers.
VERSION
Version 0.0.1
SYNOPSIS
use Algorithm::ToNumberMunger;
# one munger from a spec hash
my $code = Algorithm::ToNumberMunger->build(
{ munger => 'enum', map => { GET => 0, POST => 1, PUT => 2 } },
);
my $n = $code->('POST'); # 1
# a whole table of them at once, from a 'field => spec' hash
my $by_tag = Algorithm::ToNumberMunger->build_all(
\%mungers,
);
my $row_value = $by_tag->{method}->($raw{method});DESCRIPTION
Many numeric pipelines -- anomaly detectors, feature stores, CSV loaders -- want every column to be a number, but the values they are handed are not always numbers to begin with: an HTTP method is a string, a timestamp is a formatted date, a high-cardinality label wants bucketing. An input munger turns such a raw value into a single number. Munging happens on the input side, before a row is stored.
Mungers are declared as a plain data spec -- a hash naming a built-in munger and carrying that munger's parameters -- so a table of them can be read straight out of JSON or a config file:
{
"method": { "munger": "enum", "map": { "GET": 0, "POST": 1 } },
"bytes": { "munger": "log", "offset": 1 },
"label": { "munger": "hash", "buckets": 1024 }
}Any field without an entry is raw and is passed through unchanged; this module is only concerned with fields that name a munger.
This class does not read or write files. It compiles a spec into a closure that maps one raw value to one number, so a caller can build its mungers once from configuration and then apply them per row with no re-parsing. All configuration errors are caught at build time; the returned closure only croaks on genuinely un-mungeable input.
CLASS METHODS
build
my $code = ...->build( \%spec );
my $code = ...->build( \%spec, $tag_name ); # $tag_name only sharpens errorsCompile a single munger spec into a coderef. %spec must contain a munger key naming one of the "BUILT-IN MUNGERS"; the remaining keys are that munger's parameters. Croaks on an unknown munger name or an invalid parameter set. The optional second argument is only used to make error messages point at a tag.
build_all
my $by_tag = ...->build_all( $info->{mungers} );Compile a whole mungers hash (tag name => spec) into a hash of tag name => coderef. A false/absent argument yields an empty hashref (every tag is raw). Croaks if any spec is invalid, naming the offending tag.
compile
my $plan = ...->compile( tags => \@tags, mungers => $info->{mungers} );
my $row = $plan->apply_named( \%named_input ); # numbers, in tags orderCompile a set's tags and (optional) mungers into a plan object that maps one input record to a fully-numeric row in tag order. Unlike "build_all" (which just compiles each spec in isolation), compile understands the whole set:
a scalar munger, keyed by its output tag, fills that one column; its input is read from the tag's own name, or from
from => 'other'to alias a source field;an expanding munger, keyed by any label and carrying
into => [tag, ...], reads one source (from, defaulting to the label) and fills several columns at once -- this is how a single timestamp becomes both asin/cospair without the two ever drifting apart (see "datetime");a combining munger, keyed by its output tag and carrying a
fromlist (from => ['bytes_out', 'bytes_in']), reads several source fields and fills that one column -- this is how a ratio becomes a single feature without precomputing it upstream (see "ratio" and "combine"). The sources are raw input fields, not other (possibly munged) columns;every remaining tag is raw and passed through unchanged.
Coverage is validated up front: compile croaks if two mungers write the same column, if an into names a column not in tags, if a munger key is neither a tag nor an expander, if an expander's output count does not match its into, or if a from list is given to a munger that cannot combine inputs. The returned plan has two methods, both returning an arrayref of numbers in tags order: apply_named(\%hash) (keyed by field name, the only form that supports expanders and combiners) and apply_positional(\@row) (positional; croaks if the set has any expanding or combining munger, since a shared or combined source cannot be expressed by position).
known_mungers
my @names = ...->known_mungers;The sorted list of built-in munger names this version understands.
has_munger
if ( ...->has_munger('enum') ) { ... }True if the named munger is built in.
BUILT-IN MUNGERS
Every munger returns a plain number and, where the input cannot be interpreted, croaks -- the Writer would reject a non-numeric field anyway, so failing at the munger gives a better message. Parameters are validated when the munger is built, not per row.
enum
{ munger => 'enum', map => { GET => 0, POST => 1 }, default => -1 }Categorical string to number via an explicit map. All map values must be numeric. Without a default, an unmapped input croaks; with one, unmapped inputs (including undef) yield the default.
frozen_freq_map
{ munger => 'frozen_freq_map', counts => { jpg => 40213, exe => 12, scr => 3 },
total => 67560 }
# defaults: mode => 'neg_log_prob', smoothing => 1, unseen => 'rare'Frequency-encoding from a precomputed, frozen count table: the rarer a value was when the table was built, the more anomalous it scores. This is enum's cousin -- a value-to-number map -- except the numbers are derived from observed counts rather than hand-authored, with the smoothing and unseen-value policy that "rare = interesting" needs. It stays a stateless munger: the table is computed offline and shipped in info.json; this class only applies it.
counts maps each value to how many times it was seen. total is the overall observation count; it defaults to the sum of counts, but may be given explicitly and larger so you can prune the long tail out of counts while still computing correct probabilities. The emitted number depends on mode:
neg_log_prob(default) - self-information-ln(prob): rare values score high, common ones low. This is the axis "rare = interesting" describes and what an Isolation Forest splits on most naturally.freq- the probability itself,(count + smoothing) / denom.log_count-ln(1 + count), the count with its heavy tail tamed.count- the raw count.
Probabilities use add-one style smoothing (default 1), treating "unseen" as one aggregate bucket: prob(v) = (count + smoothing) / (total + smoothing*(V+1)) where V is the number of listed values. unseen controls what a value absent from the table maps to -- 'rare' (default) emits that value under the current mode as if it had been seen zero times (for neg_log_prob/freq this is the smoothed unseen bucket, for count/log_count it is 0), or a number to force a fixed default. Because an unseen value is usually the very thing you are hunting, mapping it to "maximally rare" rather than erroring is the point.
frozen_freq_map only suits bounded, moderate-cardinality columns (extensions, vendor classes, named pipes, keyboard layouts, link addresses): the table lives in info.json, so a huge one bloats every read -- building one past $Algorithm::ToNumberMunger::FROZEN_FREQ_MAP_WARN_KEYS values (default 10000) warns. For unbounded cardinality (JA3, full user-agents) use "hash" instead, which needs no table but keeps only decorrelation, not commonness.
http_enum
{ munger => 'http_enum' }
{ munger => 'http_enum', strict => 1 }Collapse an HTTP status code to its class: 1xx to 1, 2xx to 2, 3xx to 3, and so on (i.e. int(code / 100)). This is the usual bucketing for an HTTP status column -- the forest cares far more about "was this a 4xx vs a 2xx" than about 403 vs 404, and it keeps the feature low-cardinality without having to spell out every code in an enum map. The input must be numeric.
By default any numeric input is bucketed, so a bogus 700 would quietly become 7. With a true strict, inputs outside the valid HTTP status range (100-599) croak instead, so a malformed code is caught at write time rather than smuggled into the model as a spurious class.
smtp_enum
{ munger => 'smtp_enum' }
{ munger => 'smtp_enum', strict => 1 }The SMTP counterpart of "http_enum": collapse an SMTP reply code to its leading digit (int(code / 100)), since that digit is the reply's meaning -- 2yz completion, 3yz intermediate, 4yz transient failure, 5yz permanent failure. As with http_enum this keeps the column low-cardinality and lets the forest weigh "a 5xx where a 2xx was expected" without enumerating every code.
With a true strict, inputs outside the valid SMTP reply range (200-599) croak. SMTP never issues 1yz replies in practice (no command permits a positive-preliminary reply), so the strict floor is 200 rather than http_enum's 100.
sip_enum
{ munger => 'sip_enum' }
{ munger => 'sip_enum', strict => 1 }The SIP counterpart of "http_enum": collapse a SIP status code to its leading digit (int(code / 100)). SIP reuses HTTP's class scheme but adds a sixth class -- 1xx provisional, 2xx success, 3xx redirection, 4xx client error, 5xx server error, 6xx global failure.
With a true strict, inputs outside the valid SIP status range (100-699) croak. The ceiling is 699 rather than http_enum's 599 precisely because of that 6xx global-failure class.
ftp_enum
{ munger => 'ftp_enum' }
{ munger => 'ftp_enum', strict => 1 }The FTP counterpart of "http_enum", for FTP reply codes: int(code / 100), bucketing into 1yz-5yz. With a true strict, inputs outside 100-599 croak.
rtsp_enum
{ munger => 'rtsp_enum' }
{ munger => 'rtsp_enum', strict => 1 }The RTSP counterpart of "http_enum". RTSP (RFC 2326) deliberately reuses HTTP's status scheme, so codes collapse to their leading digit the same way. With a true strict, inputs outside 100-599 croak.
nntp_enum
{ munger => 'nntp_enum' }
{ munger => 'nntp_enum', strict => 1 }The NNTP counterpart of "http_enum", for NNTP (RFC 3977) reply codes, which follow the SMTP convention -- 1xx informational, 2xx success, 3xx send-more-input, 4xx transient failure, 5xx permanent failure. Unlike SMTP, NNTP does issue 1xx replies (help text, capability lists), so the strict floor is 100 rather than smtp_enum's 200. With a true strict, inputs outside 100-599 croak.
dict_enum
{ munger => 'dict_enum' }
{ munger => 'dict_enum', strict => 1 }The DICT counterpart of "http_enum", for DICT protocol (RFC 2229) status codes, which use the SMTP-style code classes. With a true strict, inputs outside 100-599 croak.
gemini_enum
{ munger => 'gemini_enum' }
{ munger => 'gemini_enum', strict => 1 }Like "http_enum" but for the Gemini protocol, whose status codes are two digits -- 1x input expected, 2x success, 3x redirect, 4x temporary failure, 5x permanent failure, 6x client certificate required -- so the class is int(code / 10). With a true strict, inputs outside 10-69 croak.
mgcp_enum
{ munger => 'mgcp_enum' }
{ munger => 'mgcp_enum', strict => 1 }The MGCP counterpart of "http_enum", for MGCP (RFC 3435) response codes: int(code / 100). MGCP's classes are 1xx provisional, 2xx success, 4xx transient error, 5xx permanent error, and 8xx package-specific -- there are no 6xx or 7xx codes, so the valid set has a hole in it. With a true strict, inputs outside 100-599 and outside 800-899 croak. (That hole is why this is a hand-written builder rather than another row of the shared status-class table, which can only express one contiguous range.)
dns_rcode_enum
{ munger => 'dns_rcode_enum' }
{ munger => 'dns_rcode_enum', default => -1 }The first of the named-map enums: like "enum", except the map is baked in from a well-known registry instead of hand-authored (and inevitably typo'd). All named-map enums share the same lookup rules: names are matched case-insensitively; where the emitted numbers are the protocol's own wire encoding (as here -- rcode 3 is NXDOMAIN), a numeric input is passed through unchanged, so mixed feeds (one tool logs NXDOMAIN, another logs 3) land in one consistent column; and an unmapped value croaks unless the spec supplies a numeric default. As with enum, an unrecognized value is often exactly the anomaly worth keeping, so default => -1 is the usual escape hatch.
This one maps DNS RCODE names to their IANA values: NOERROR 0, FORMERR 1, SERVFAIL 2, NXDOMAIN 3, NOTIMP 4 (alias NOTIMPL), REFUSED 5, YXDOMAIN 6, YXRRSET 7, NXRRSET 8, NOTAUTH 9, NOTZONE 10, DSOTYPENI 11, and the extended rcodes BADVERS/BADSIG 16, BADKEY 17, BADTIME 18, BADMODE 19, BADNAME 20, BADALG 21, BADTRUNC 22, BADCOOKIE 23.
dns_qtype_enum
{ munger => 'dns_qtype_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum"; numeric inputs pass through) mapping DNS RR type names to their IANA numbers: A 1, NS 2, CNAME 5, SOA 6, NULL 10, PTR 12, MX 15, TXT 16, AAAA 28, SRV 33, NAPTR 35, DS 43, RRSIG 46, DNSKEY 48, TLSA 52, SVCB 64, HTTPS 65, AXFR 252, ANY (or *) 255, URI 256, CAA 257, and the rest of the commonly-observed registry. The query-type mix is a classic DNS-tunneling feature -- TXT/NULL-heavy traffic where A/AAAA is normal.
syslog_severity_enum
{ munger => 'syslog_severity_enum' }Named-map enum (lookup rules as "dns_rcode_enum"; numeric inputs pass through) mapping syslog severity names to their RFC 5424 codes: emerg 0 (alias panic), alert 1, crit 2, err 3 (alias error), warning 4 (alias warn), notice 5, info 6 (alias informational), debug 7. Genuinely ordinal -- lower is more severe -- so a threshold split on it is meaningful.
syslog_facility_enum
{ munger => 'syslog_facility_enum' }Named-map enum (lookup rules as "dns_rcode_enum"; numeric inputs pass through) mapping syslog facility names to their RFC 5424 codes: kern 0, user 1, mail 2, daemon 3, auth 4 (alias security), syslog 5, lpr 6, news 7, uucp 8, cron 9, authpriv 10, ftp 11, ntp 12, audit 13, alert 14, clock 15, and local0-local7 16-23.
ip_proto_enum
{ munger => 'ip_proto_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum"; numeric inputs pass through) mapping IP protocol names to their IANA protocol numbers: icmp 1, igmp 2, ipip 4 (alias ipencap), tcp 6, egp 8, udp 17, dccp 33, ipv6 41, rsvp 46, gre 47, esp 50, ah 51, icmpv6 58 (alias ipv6-icmp), ospf 89, pim 103, sctp 132, udplite 136. The map is frozen here rather than delegated to getprotobyname so a value munges to the same number on every host.
tls_version_enum
{ munger => 'tls_version_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") mapping a TLS/SSL protocol version name to an ordinal: SSLv2 0, SSLv3 1, TLSv1 2, TLSv1.1 3, TLSv1.2 4, TLSv1.3 5, with the common spelling variants (ssl3, tls1.2, ...) accepted. Ordinal so "older than expected" is a monotone feature a threshold split can express. Because these ordinals are this module's invention rather than a wire encoding, numeric inputs are not passed through -- a 1.2 would land on the wrong scale -- and croak like any other unmapped value (or take the default).
http_method_enum
{ munger => 'http_method_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the registered HTTP request methods: GET 0, HEAD 1, POST 2, PUT 3, DELETE 4, CONNECT 5, OPTIONS 6, TRACE 7, PATCH 8. HTTP has no numeric method encoding, so these are unordered ordinals of this module's invention (a canonical map beats every set inventing its own numbering) and numeric inputs are not passed through. An unlisted -- possibly probing -- method croaks unless a default is given, and that unlisted-method signal is often the interesting one.
sip_method_enum
{ munger => 'sip_method_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the SIP request methods: INVITE 0, ACK 1, BYE 2, CANCEL 3, REGISTER 4, OPTIONS 5, PRACK 6, SUBSCRIBE 7, NOTIFY 8, PUBLISH 9, INFO 10, REFER 11, MESSAGE 12, UPDATE 13. Like "http_method_enum" these are ordinals of this module's invention, so numeric inputs are not passed through.
dhcp_msgtype_enum
{ munger => 'dhcp_msgtype_enum' }Named-map enum (lookup rules as "dns_rcode_enum"; numeric inputs pass through) mapping DHCP message-type names to their option-53 values: DISCOVER 1, OFFER 2, REQUEST 3, DECLINE 4, ACK 5, NAK 6, RELEASE 7, INFORM 8 -- each also accepted with the DHCP prefix (DHCPDISCOVER, ...) that most tooling logs.
app_proto_enum
{ munger => 'app_proto_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for Suricata's app_proto field -- the detected application-layer protocol on a flow or alert (http, dns, tls, ssh, smtp, dcerpc, quic, ...), including failed and unknown, which are usually the very rows worth keeping. These are unordered ordinals of this module's invention (Suricata logs a string, not a number), so numeric inputs are not passed through. ssl is accepted as an alias for tls and ikev2 for ike. This is distinct from "ip_proto_enum", which numbers the L4 protocol (tcp/udp/...) rather than the app layer riding on it.
tcp_state_enum
{ munger => 'tcp_state_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") mapping the TCP state machine, as Suricata logs it under flow.tcp.state, to an ordinal along the connection lifecycle: none 0, syn_sent 1, syn_recv 2, established 3, fin_wait1 4, fin_wait2 5, closing 6, time_wait 7, close_wait 8, last_ack 9, closed 10. Ordinal so "further along teardown than expected" is a monotone feature a threshold split can express. Being ordinals of our own invention, numeric inputs are not passed through.
flow_state_enum
{ munger => 'flow_state_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for Suricata's flow.state: new 0, established 1, closed 2, bypassed 3, local_bypass 4 -- roughly ordinal along the flow lifecycle. Numeric inputs are not passed through.
flow_reason_enum
{ munger => 'flow_reason_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for Suricata's flow.reason, why a flow was logged out: timeout 0, forced 1, shutdown 2, unknown 3. Numeric inputs are not passed through.
suricata_action_enum
{ munger => 'suricata_action_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for Suricata's alert.action and the related rule/drop actions: allowed 0, blocked 1, pass 2, drop 3, reject 4, alert 5. In IDS mode the field is allowed/blocked; the rule-action names are accepted too for IPS feeds and drop events. Numeric inputs are not passed through.
postfix_status_enum
{ munger => 'postfix_status_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for Postfix's delivery status= disposition, numbered in a rough sent-to-failed severity order so a threshold split is meaningful: sent 0, deferred 1, bounced 2, expired 3, deliverable 4, undeliverable 5, hold 6, discard 7, filtered 8, reject 9, softbounce 10. Stock delivery agents emit only sent/deferred/bounced/expired; deliverable/undeliverable come from address verification (verify), and the remainder cover HOLD/DISCARD actions and values common log normalizers emit. Being labels of this module's numbering, numeric inputs are not passed through.
spf_result_enum
{ munger => 'spf_result_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for an SPF check result (RFC 7208), as logged by policyd-spf or carried in an Authentication-Results header, numbered pass-to-fail: pass 0, neutral 1, none 2, softfail 3, fail 4, temperror 5, permerror 6. The older spellings error (for temperror) and unknown (for permerror) are accepted as aliases. Numeric inputs are not passed through.
dkim_result_enum
{ munger => 'dkim_result_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for a DKIM verification result (RFC 8601), as logged by opendkim or carried in an Authentication-Results header, numbered pass-to-fail: pass 0, neutral 1, none 2, policy 3, fail 4, temperror 5, permerror 6. The older spellings error (for temperror) and unknown (for permerror) are accepted as aliases. Numeric inputs are not passed through.
dmarc_result_enum
{ munger => 'dmarc_result_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for a DMARC evaluation result (RFC 7489 / RFC 8601), as logged by opendmarc or carried in an Authentication-Results header: pass 0, none 1, fail 2, temperror 3, permerror 4, and opendmarc's bestguesspass 5. This is the DMARC result (did the message pass alignment), not the policy disposition (none/quarantine/reject) -- for that, use a plain "enum". Numeric inputs are not passed through.
sasl_mech_enum
{ munger => 'sasl_mech_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the SASL authentication mechanism -- as Dovecot logs mech=, Postfix logs sasl_method=, and submission/IMAP/POP3 auth report -- numbered in a rough weakest-to-strongest order so the ordinal carries a little signal on its own: the cleartext and anonymous mechanisms sort low, the legacy challenge-response ones (cram-md5, digest-md5, ntlm, ...) in the middle, then srp/scram-*, the OAuth/federated tokens, and finally the Kerberos/GSS and certificate (external) mechanisms. About two dozen mechanisms are baked in, covering the IANA registry plus the ubiquitous non-registered login, xoauth2, and apop. Being ordinals of this module's numbering, numeric inputs are not passed through; an unlisted mechanism croaks unless a numeric default is given.
If you would rather the number carry no implied gradient, see "sasl_mech_iana_enum", which numbers the same set alphabetically.
sasl_mech_iana_enum
{ munger => 'sasl_mech_iana_enum', default => -1 }The nominal counterpart of "sasl_mech_enum": the same set of SASL mechanisms (the two share one list, so they can never cover different mechanisms), but numbered alphabetically rather than by strength -- a purely categorical encoding for when a strength gradient would be a misleading feature. Lookup rules and the no-numeric-passthrough behaviour are as "sasl_mech_enum".
http_version_enum
{ munger => 'http_version_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") mapping the HTTP protocol version to an ordinal: HTTP/0.9 0, HTTP/1.0 1, HTTP/1.1 2, HTTP/2.0 3, HTTP/3.0 4. The access-log spelling (HTTP/1.1), the bare number (1.1), and the ALPN/shorthand forms (h2, h2c, h3) are all accepted, so an Apache/nginx %H field and a Suricata http.protocol land in one column. Ordinal so "older than expected" (a 0.9/1.0 request from a scanner) is a monotone feature a threshold split can express. Because these are ordinals of our own numbering -- and a logged 2 denotes version 2.0, not the integer two -- numeric inputs are not passed through.
spamassassin_autolearn_enum
{ munger => 'spamassassin_autolearn_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for SpamAssassin's autolearn= field: no 0, ham 1, spam 2, disabled 3, failed 4, unavailable 5, unknown 6. The numbering is essentially nominal (the Bayes auto-learn outcome is a category, not a scale), arranged only so the "nothing was learned" states cluster away from the ham/spam ones. The spam score is already a number for "num", and the spam/ham verdict a "bool"; this covers the one autolearn field neither derives. Numeric inputs are not passed through.
rspamd_action_enum
{ munger => 'rspamd_action_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for rspamd's action, numbered by increasing severity so the ordinal is a usable feature on its own: no action 0, greylist 1, add header 2, rewrite subject 3, soft reject 4, reject 5. Both the space and underscore spellings rspamd emits (no action/no_action, add header/add_header, ...) are accepted. Numeric inputs are not passed through.
ssh_auth_method_enum
{ munger => 'ssh_auth_method_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the OpenSSH authentication method logged by sshd (the word after Accepted/Failed, or the method= field): none 0, password 1, keyboard-interactive 2, hostbased 3, publickey 4, gssapi-with-mic 5, gssapi-keyex 6, with bare gssapi aliased to gssapi-with-mic. Numbered roughly weakest-to-strongest so "weaker credential than expected" is a monotone feature; the ordering is a judgement call, not a registry. Numeric inputs are not passed through.
amavis_category_enum
{ munger => 'amavis_category_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the amavisd-new content category, ordered clean-to-worst: clean 0, oversized 1, unchecked 2, spammy 3, spam 4, badheader 5, banned 6, infected 7, mtablocked 8. The hyphenated spellings (bad-header, mta-blocked) and the legacy virus (for infected) are accepted as aliases. The Passed/Blocked action itself is a "bool"; this is the finer-grained reason. Numeric inputs are not passed through.
systemd_result_enum
{ munger => 'systemd_result_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for a systemd unit result (as in Failed with result 'timeout'): success 0, protocol 1, timeout 2, exit-code 3, signal 4, core-dump 5, watchdog 6, start-limit-hit 7, oom-kill 8, resources 9. The underscore spellings (exit_code, core_dump, start_limit_hit, oom_kill) are accepted as aliases. Numeric inputs are not passed through.
clamav_result_enum
{ munger => 'clamav_result_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for a ClamAV per-target verdict as logged by clamd / clamav-milter: OK 0, FOUND 1, ERROR 2. Small but exactly the signal worth flagging (a FOUND line). Numeric inputs are not passed through.
kerberos_etype_enum
{ munger => 'kerberos_etype_enum', default => -1 }Named-map enum for the Kerberos ticket encryption type -- Windows events 4768/4769 (logged as hex, 0x17) and any other AD/Kerberos source. The values are the RFC 3961 etype numbers, which are the wire encoding, so (unlike most of the enums here) a decimal input passes through unchanged: the map exists only to resolve the hex spellings (0x17 => 23, 0x12 => 18, ...) and the RFC/MIT names (rc4-hmac/rc4/arcfour-hmac => 23, aes256-cts-hmac-sha1-96/aes256 => 18, aes128 => 17, des3 => 16, des-cbc-md5 => 3, des-cbc-crc => 1) onto that same number. The classic use is flagging rc4-hmac (0x17) as a downgrade/roasting signal against an aes256 baseline. Lookup is case-insensitive; an unlisted etype croaks unless a numeric default is given.
windows_integrity_level_enum
{ munger => 'windows_integrity_level_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the Windows / Sysmon process integrity level, as an ordinal: untrusted 0, low 1, medium 2, high 3, system 4 (with mediumplus folded into medium). The S-1-16-* mandatory-label SIDs Windows sometimes logs in place of the word (S-1-16-12288 => 3, ...) are accepted as aliases. Ordinal so "higher privilege than expected" is a monotone feature. Numeric inputs are not passed through.
windows_logon_status_enum
{ munger => 'windows_logon_status_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the NTSTATUS sub-status on a failed Windows logon (events 4625/4776), collapsed to a compact reason category rather than the raw 32-bit code: 0xC0000064 (no such user) 0, 0xC000006A (bad password) 1, 0xC000006D (generic bad user/pass) 2, 0xC000006F (outside hours) 3, 0xC0000070 (workstation restriction) 4, 0xC0000071 (password expired) 5, 0xC0000072 (disabled) 6, 0xC0000193 (account expired) 7, 0xC0000133 (clock skew) 8, 0xC0000224 (must change password) 9, 0xC0000234 (locked out) 10, 0xC000015B (logon type not granted) 11. Keys are the hex codes exactly as logged (matched case-insensitively); only the common logon subset is baked in, so an unlisted code croaks unless a numeric default is given. Numeric inputs are not passed through.
windows_impersonation_level_enum
{ munger => 'windows_impersonation_level_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the event 4624 impersonation level, ordered by reach: anonymous 0, identification 1, impersonation 2, delegation 3. The %%1832 / %%1833 message tokens Windows often emits in place of the words (Identification / Impersonation) are accepted as aliases. Numeric inputs are not passed through.
aad_signin_error_enum
{ munger => 'aad_signin_error_enum', default => -1 }Named-map enum for the Azure AD / Entra sign-in ResultType error code, collapsed to a compact reason category rather than the raw code: 0 (success) 0, invalid-password (50126, 50056) 1, no-such-user (50034) 2, disabled (50057) 3, locked / smart-lockout (50053) 4, password-expired (50055, 50144) 5, MFA-required (50074, 50076, 50079) 6, MFA-failed (500121, 50158) 7, blocked-by-conditional-access (53003, 53000, 53001, 530032) 8, session-expired (50173) 9. Although ResultType is already numeric, the code space is huge and sparse and its magnitude carries no signal -- this maps the common codes onto a handful of meaningful buckets (and, unlike "hash", keeps related codes together). Keys are the codes as logged; only the common subset is baked in, so an unlisted code croaks unless a numeric default is given. Because the output is a category of our own numbering, numeric inputs are not passed through.
risk_level_enum
{ munger => 'risk_level_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the Entra Identity Protection riskLevel as an ordinal: none 0, low 1, medium 2, high 3. hidden and unknownFutureValue are left to the default (they are not points on the scale). Numeric inputs are not passed through.
aws_principal_type_enum
{ munger => 'aws_principal_type_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the CloudTrail userIdentity.type: Root 0, IAMUser 1, AssumedRole 2, FederatedUser 3, SAMLUser 4, WebIdentityUser 5, Directory 6, IdentityCenterUser 7, AWSAccount 8, AWSService 9, Unknown 10. The numbering is nominal (distinct stable numbers, not a scale); Root is the value you actually alert on. Numeric inputs are not passed through.
aad_client_app_enum
{ munger => 'aad_client_app_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the Azure AD ClientAppUsed, numbered so the modern clients sort low (Browser 0, Mobile Apps and Desktop clients 1) and the legacy-auth protocols -- which cannot satisfy MFA -- sort high (Exchange ActiveSync, IMAP4, POP3, Authenticated SMTP, MAPI Over HTTP, Exchange Web Services, Exchange Online PowerShell, AutoDiscover, Offline Address Book, Other clients, from 2 up). A ">= 2 means legacy auth" threshold is the intended feature. imap/pop/mapi short forms are accepted as aliases. Numeric inputs are not passed through.
risk_state_enum
{ munger => 'risk_state_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the Entra Identity Protection riskState: none 0, confirmedSafe 1, remediated 2, dismissed 3, atRisk 4, confirmedCompromised 5. Numeric inputs are not passed through.
vpc_flow_log_status_enum
{ munger => 'vpc_flow_log_status_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the AWS VPC Flow Logs log-status: OK 0, NODATA 1, SKIPDATA 2. (The per-flow ACCEPT/REJECT action is a plain "bool"; this is the capture-health field.) Numeric inputs are not passed through.
aws_event_type_enum
{ munger => 'aws_event_type_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the CloudTrail eventType: AwsApiCall 0, AwsServiceEvent 1, AwsConsoleAction 2, AwsConsoleSignIn 3, AwsCloudTrailInsight 4. AwsConsoleSignIn is the one worth flagging. Numeric inputs are not passed through.
conditional_access_result_enum
{ munger => 'conditional_access_result_enum', default => -1 }Named-map enum (lookup rules as "dns_rcode_enum") for the Azure AD sign-in conditionalAccessStatus: success 0, notApplied 1, notEnabled 2, reportOnly 3, failure 4. Numeric inputs are not passed through.
bool
{ munger => 'bool' } # Perl truthiness -> 1/0
{ munger => 'bool', true => [ 'yes', 'Y', '1', 'true' ] }Coerce to 1 or 0. With a true list, only those (string-compared) values are 1; otherwise ordinary Perl truthiness is used.
length
{ munger => 'length' }The character length of the stringified input, undef counting as 0 (an absent value is a zero-length one -- e.g. an SNI-absent TLS record). This is the cheap shape feature behind every *_length column (domain, URL, filename, SNI, hostname, ...): tunneling and generated names run long, so raw length is a surprisingly strong corroborator next to "entropy". Length is counted in characters, not bytes, so a multi-byte name is measured as a human would read it; use "entropy" (which is byte-oriented) when you want per-symbol randomness.
entropy
{ munger => 'entropy' }Shannon entropy of the input string, in bits per symbol -- i.e. -sum(p*log2(p)) over the frequencies of its bytes. This is the single most common feature in the pipeline (DGA domains, randomized filenames, forged User-Agents, generated SNIs / hostnames / principal names), because machine-generated strings spread their characters far more evenly than human-chosen ones and so score high, while a real word scores low. An empty string is 0; the maximum is 8 (every byte value equally likely).
Entropy is computed over the string's UTF-8 bytes (matching "hash"), so the value is well-defined regardless of the scalar's internal encoding flag. Like hash, this munger is XS-accelerated -- a per-byte histogram plus a log per distinct byte -- with a pure-Perl fallback that produces identical values; $Algorithm::ToNumberMunger::HAVE_XS says which is in use.
ngram
{ munger => 'ngram', counts => { th => 152, he => 128, in => 94, ... } }
# defaults: smoothing => 1, fold_case => 1; n is inferred from the keysMean per-gram surprisal of the input string against a precomputed, frozen n-gram count table: sum(-ln p(gram)) / gram_count, each gram's probability smoothed exactly as in "frozen_freq_map". This is frozen_freq_map's sequential cousin and the strongest single gibberish detector: "entropy" misses pronounceable generated names and is unreliable on short strings, while an n-gram score against (say) hostname bigram statistics catches both -- real words ride the common bigrams and score low, generated names keep hitting rare ones and score high. Dividing by the gram count keeps scores comparable across lengths.
counts maps each n-gram to how often it was observed when the table was built; all keys must be the same length, and that length is n (bigrams are the usual choice -- a 26x26 table stays tiny in info.json; past $FROZEN_FREQ_MAP_WARN_KEYS entries it warns like frozen_freq_map). total defaults to the sum of counts and may be given larger to prune the tail, exactly as in frozen_freq_map. A gram absent from the table gets the smoothed unseen-bucket probability -- an unseen gram is the interesting case -- so smoothing must be > 0 (default 1). With fold_case (default on) the input is lowercased before scoring, matching the usual lowercased table. A string with no grams (shorter than n) scores 0. Grams are taken over characters, matching "length" rather than the byte-oriented entropy.
char
{ munger => 'char', class => 'non_alnum', mode => 'ratio' }
{ munger => 'char', class => 'non_ascii' } # mode defaults to countCount the characters of the input that fall in a named class, either as a raw count (default) or, with mode => 'ratio', as a fraction of the string's length (0 for an empty string). This is the injection / obfuscation detector behind columns like url_non_alnum (a ratio, so it stays independent of length) and filename_non_ascii (a count): payloads and homoglyph tricks are dense with punctuation, percent-encoding, or non-ASCII where normal input is not. Counting is over characters, so non_ascii means codepoints above 127.
Recognised classes: alnum / non_alnum, ascii / non_ascii, digit, alpha, upper, lower, vowel, consonant, xdigit, space, punct. vowel and consonant are the ASCII letters (y counting as a consonant) -- a vowel/consonant ratio is a DGA corroborator that catches consonant-heavy random strings entropy alone underrates; xdigit is 0-9a-fA-F, dense in encoded payloads.
run
{ munger => 'run', class => 'consonant' }
{ munger => 'run', class => 'digit' }The length of the longest unbroken run of characters in a named class -- the same class names "char" recognises. Where char counts how many such characters occur in total, run measures how tightly they clump: the longest consonant run and longest digit run are staple generated-name (DGA) features that neither total counts nor "entropy" capture, because a real word breaks its consonants up with vowels while a random string will happily emit six in a row. An empty or undef input is 0.
count
{ munger => 'count', of => '/' } # url_path_depth, topic_depth
{ munger => 'count', of => '.', plus => 1 } # label_count (dots + 1)Count non-overlapping occurrences of a literal substring of in the input, optionally adding a constant plus. This is the segment/depth feature behind url_path_depth and topic_depth (count of `/`) and label_count (dots plus one). of is matched literally, not as a pattern, so . means a literal dot.
match
{ munger => 'match', pattern => '^xn--' } # punycode label
{ munger => 'match', pattern => '%[0-9A-Fa-f]{2}', mode => 'count' }Match the input against a Perl regular expression pattern: 1/0 under the default mode => 'bool', or the number of non-overlapping matches with mode => 'count'. A true ignore_case makes the match case-insensitive. This is the catch-all shape test behind flags like "is this label punycode" or "is the Host an IP literal", and counters like percent-escapes in a URL -- anything "char" and "count" are not expressive enough for. The pattern is compiled at build time, so a broken one fails at write_info rather than per row.
Trust note: a pattern cannot execute code (Perl requires use re 'eval' for that, which this module does not enable), but a pathological pattern can still backtrack catastrophically and stall a writer. Treat munger specs -- like the rest of info.json -- as configuration from a trusted operator, not as untrusted input.
bucket
{ munger => 'bucket', bounds => [ 1024, 49152 ] } # dest_port classesMap a number to a bucket index by ascending bounds: the result is how many bounds the value is greater than or equal to. With bounds => [1024, 49152] a value under 1024 is 0 (well-known), 1024-49151 is 1 (registered), and 49152+ is 2 (ephemeral) -- the classic port classing, where the literal port number is meaningless to a threshold split but the class is a real signal. bounds must be strictly ascending; N bounds yield indices 0..N.
This generalises the *_enum status-class mungers, which are the special case of bucketing a reply code by its leading digit.
quantile
{ munger => 'quantile', bounds => [ 40, 180, 460, 2200, 64000 ] }Piecewise-linear ECDF: map a number onto [0, 1] by where it falls among ascending bounds taken from the training data's quantiles (e.g. its min / p25 / p50 / p75 / max). Values at or below the first bound map to 0, at or above the last to 1, and anything between two adjacent bounds interpolates linearly between their positions. This is "bucket"'s continuous sibling and the heavy-tail normaliser to reach for when "log" is not enough and "zscore" would let one outlier stretch the whole scale: after the transform the training distribution is roughly uniform, so a forest threshold split lands anywhere in it with equal ease. bounds must be strictly ascending with at least two values; like zscore, the parameters are supplied rather than learned, so munging stays stateless.
scale
{ munger => 'scale', min => 0, max => 1000, clamp => 1 }Min-max normalisation: (v - min) / (max - min), mapping [min, max] onto [0, 1]. min and max must differ. With a true clamp, results are pinned into [0, 1] so out-of-range inputs cannot escape the unit interval.
zscore
{ munger => 'zscore', mean => 42.0, std => 7.5 }Standardise: (v - mean) / std. std must be non-zero. The mean/std are supplied (this module does not learn them) so munging stays stateless and a row can be munged in isolation.
log
{ munger => 'log' } # natural log
{ munger => 'log', offset => 1 } # log1p-style, so 0 is allowed
{ munger => 'log', base => 10, offset => 1 }Logarithm of v + offset. Heavy-tailed counts (bytes, durations) compress well under a log, which keeps a few huge values from dominating the forest. offset (default 0) shifts the input so zeros/small values are representable; the shifted value must be strictly positive or the input croaks. base defaults to natural log.
clamp
{ munger => 'clamp', min => 0 }
{ munger => 'clamp', min => 0, max => 65535 }Pass the number through, pinned into [min, max]. Either bound may be omitted to clamp on one side only. Unlike scale this does not rescale; it only caps outliers before they reach the model.
num
{ munger => 'num', base => 16 } # '0x1a' or '1a' -> 26
{ munger => 'num' } # plain numeric coercionParse a string as a number in base (2-36, default 10). Base 10 simply validates and numifies. Other bases accept the digits 0-9a-z below the base, case-insensitively, an optional leading -, and the conventional prefix for that base (0x for 16, 0b for 2, 0o for 8). Plenty of tooling logs flag words and IDs in hex (0x2f), which the Writer would reject as non-numeric; this munger is the bridge. Croaks on anything that is not a clean number in the chosen base.
ratio
# 'io_ratio' is a tag; bytes_out and bytes_in are input fields
"io_ratio": { "munger": "ratio", "from": ["bytes_out", "bytes_in"] }
{ munger => 'ratio', from => [qw(bytes_out bytes_in)], zero => -1 }First source divided by the second: with from => [a, b] the column gets a / b. Asymmetry between two counters is a classic feature the counters alone cannot express -- bytes out over bytes in flags exfiltration, requests over responses flags scanning -- and the division has to happen at munge time because a forest split only ever sees one column. A zero denominator yields zero (default 0) instead of dying, since "nothing came back" is a legitimate row, not bad input; pick a zero outside the ratio's normal range if you want those rows to stand out. Both inputs must be numeric.
This is a multi-input munger: it only makes sense with several sources, so it is only usable through "compile" with from as an arrayref of exactly two field names (and thus apply_named / write_named). The sources are raw input fields, not other columns.
combine
{ munger => 'combine', op => 'sum', from => [qw(bytes_in bytes_out)] }
{ munger => 'combine', op => 'max', from => [qw(req_time resp_time)] }Fold two or more numeric source fields into one column with op: sum, diff (first minus second; exactly two sources), product, min, max, or mean. The general-purpose sibling of "ratio" for when the interesting feature is a total, a gap, or an extreme across fields rather than any one field. Every input must be numeric.
Like ratio, this is a multi-input munger: only usable through "compile" with from as an arrayref of source field names.
bit
{ munger => 'bit', mask => '0x12' } # SYN or ACK set?
{ munger => 'bit', mask => '0x02', mode => 'all' } # the SYN bit itself
{ munger => 'bit', mode => 'popcount' } # how many flags at all
{ munger => 'bit', mask => '0x0f', mode => 'value' } # low nibble, 0-15
{ munger => 'bit', mask => '0x02', base => 16 } # Suricata tcp_flags "1b"Bit-level features from an integer flags word (TCP flags, DNS header flags, protocol option words): the raw word is meaningless to a threshold split, but individual bits and bit counts are real signals. The input must be a non-negative integer, in decimal or 0x hex (so a logged 0x12 works as-is); mask may be written either way too.
Set base => 16 to read the input as bare hexadecimal with no 0x prefix -- Suricata logs tcp.tcp_flags (and tcp_flags_ts/tcp_flags_tc) as e.g. "1b", which is otherwise ambiguous with decimal. A 0x prefix on the input is still accepted under base => 16. mask is always written in decimal or 0x hex regardless of base. Modes:
any(default) -1if any bit ofmaskis set in the value.all-1only if every bit ofmaskis set.value- the masked bits, shifted down to the mask's lowest set bit:mask => '0x0f'extracts the low nibble as0-15.popcount- the number of set bits invalue & mask;maskis optional here and defaults to all bits. An abnormal flag count (a Christmas-tree packet) is anomalous even when each individual bit is common.
mask is required (and must be non-zero) for every mode except popcount.
ip_class
{ munger => 'ip_class' }
{ munger => 'ip_class', default => -1 }Collapse an IPv4 or IPv6 address to its address-space class -- to addresses what the status-class enums are to reply codes: the literal address is high-cardinality noise, but "an internal host suddenly talking multicast" is a class-level signal. Classes and their emitted numbers:
0 global anything not covered below
1 private 10/8, 172.16/12, 192.168/16, 100.64/10 (CGNAT), fc00::/7 (ULA)
2 loopback 127/8, ::1
3 link_local 169.254/16, fe80::/10
4 multicast 224/4, ff00::/8
5 broadcast 255.255.255.255
6 unspecified 0.0.0.0, ::
7 reserved 0/8, 192.0.0/24, the documentation nets (192.0.2/24,
198.51.100/24, 203.0.113/24, 2001:db8::/32), benchmarking
(198.18/15), 240/4, and the 100::/64 discard prefixAn IPv4-mapped IPv6 address (::ffff:a.b.c.d) is classified as its embedded IPv4 address. An unparseable input croaks, or yields the numeric default when one is given. IPv6 parsing uses Socket's inet_pton, loaded lazily the way "datetime" loads Time::Piece. For site-specific zones (DMZ, server VLAN, guest Wi-Fi) use "cidr", which knows your networks instead of the RFCs'.
cidr
{ munger => 'cidr',
nets => [ '10.10.0.0/16', '10.20.0.0/16', '2001:db8:5::/48' ],
default => -1 }Membership in a list of CIDR networks: the result is the (0-based) index of the first net in nets containing the address -- "bucket" for address space, and the way a site encodes its own zones (DMZ vs. server VLAN vs. guest Wi-Fi) that "ip_class"'s generic RFC classes cannot know about. nets may mix IPv4 and IPv6; an address is only tested against nets of its own family. Overlapping nets are fine -- list the most specific first, since the first match wins. An input that is unparseable or in none of the listed nets croaks, or yields the numeric default when one is given (a catch-all default is the usual configuration).
datetime
{ munger => 'datetime', format => '%Y-%m-%dT%H:%M:%S', part => 'epoch' }
{ munger => 'datetime', format => '%Y-%m-%d %H:%M:%S', part => 'hour' }Parse a formatted timestamp with Time::Piece (strptime, so format is a standard strptime pattern) and extract one numeric part:
epoch(default) - seconds since the epoch.year,mon(1-12),mday(1-31),hour,min,sec.wday- day of week,0=Sunday ..6=Saturday.yday- day of year,0-based.frac_day- time of day as a fraction in[0, 1), i.e.(hour*3600 + min*60 + sec) / 86400. Handy as a cyclic-ish time-of-day feature.frac_week- position within the week as a fraction in[0, 1), the week starting Sunday to matchwday:(wday*86400 + hour*3600 + min*60 + sec) / 604800. Likefrac_daybut cycling over a week, so a weekly rhythm (weekend vs. weekday, or a Monday-morning batch) shows up as a feature.sin_day/cos_day,sin_week/cos_week- thefrac_*value mapped onto a circle,sin(2*pi*frac)andcos(2*pi*frac). Prefer these over the rawfrac_*when feeding the forest: a plain fraction has a false seam at the wrap (23:59 and 00:00 sit at opposite ends, 1 vs 0, though they are a minute apart), whereas the sin/cos pair is continuous across midnight/Sunday. Store both of a pair in two columns so the position is unambiguous.
Time features often carry the anomaly (a job that normally runs at 03:00 suddenly firing at noon, or a weekday task firing on a Sunday), which is why this is a first-class munger.
Multi-output form. A cyclic pair belongs together -- sin alone collides (sin is symmetric about its peak, so two different times map to one value) and the forest then treats distinct times as identical. To emit a pair atomically, give parts (plural) and route them to two columns with into (see "compile"):
"time_of_week": {
"munger": "datetime", "from": "timestamp",
"format": "%Y-%m-%dT%H:%M:%S",
"parts": [ "sin_week", "cos_week" ],
"into": [ "time_sin", "time_cos" ]
}The timestamp is parsed once and both columns are filled together, so they can never drift apart or be half-configured. parts and into must be the same length. (Using parts without into, or part with into, is an error.)
Performance. Two transparent accelerations, both value-identical to the plain path: a one-slot memo returns the previous result when the same stamp string repeats (the common case in bursty event streams); and when the format is built from only the six numeric codes %Y %m %d %H %M %S (once each, e.g. %Y-%m-%dT%H:%M:%S), parsing skips strptime for a compiled regex plus integer date math, falling back to strptime for any value the regex does not match or whose fields are out of range (a month 13, an hour 24, a Feb 30) -- so an invalid stamp croaks or normalizes exactly as strptime would, never silently feeding nonsense to the date math. Like strptime without a zone code, stamps are treated as UTC.
hash
{ munger => 'hash', buckets => 1024 }
{ munger => 'hash', buckets => 1024, seed => 7 }
{ munger => 'hash' } # raw 32-bit FNV-1a valueFeature hashing for high-cardinality categoricals you do not want to (or cannot) enumerate with enum. The input is stringified and run through 32-bit FNV-1a; with buckets the result is reduced modulo that many buckets ([0, buckets)), otherwise the full 32-bit hash is returned. An optional seed lets you decorrelate two hashed columns.
This is the one munger that is XS-accelerated: FNV-1a is a per-byte loop with a 32-bit modular multiply, which is slow in pure Perl and (on a 32-bit perl) fussy to get exactly right. $Algorithm::ToNumberMunger::HAVE_XS reports whether the compiled path is in use; a pure-Perl fallback (exact on a 64-bit perl) is used otherwise, and both produce identical values.
chain
# Shannon entropy of just the TLD: lowercase, keep the last dot-label
{ munger => 'chain',
steps => [ { op => 'lc' }, { op => 'split', on => '.', index => -1 } ],
then => { munger => 'entropy' } }
# a hex request id buried in a token like 'req-0x2F'
{ munger => 'chain',
steps => [ { op => 'capture', pattern => 'req-(0x[0-9a-fA-F]+)' } ],
then => { munger => 'num', base => 16 } }Run the input through a list of string pre-transforms (steps, applied in order), then hand the result to a terminal munger (then) for the actual number. Every string munger above scores the whole value; chain is how a feature targets a piece of it -- the entropy of the TLD alone, the length of the first path segment, an enum over a normalized token -- without asking the writer's caller to pre-slice its input. Each step is a hashref with an op:
lc/uc- case-fold the value.trim- strip leading and trailing whitespace.split- split on the literal separatoronand keep pieceindex(0-based; negative counts from the end, so-1is a hostname's last label). An index past either end yields the empty string.capture- match the regexpatternand keep capture groupgroup(default1). No match, or a group that did not participate, yields the empty string. A trueignore_casematches case-insensitively; the "match" trust note applies here too.replace- replace every match of the regexpatternwith the literal stringwith(default: delete the matches).ignore_caseas above.
then is a full munger spec and may be any built-in that takes one value -- including another chain. All step parameters and the terminal spec are validated at build time. An undef input enters the chain as the empty string; whether an empty result is acceptable is the terminal's call (entropy and length score it 0, num croaks).
The multi-output form works too: put into on the chain and the parts on the terminal, e.g. trim a sloppy timestamp before a "datetime" sin/cos expansion.
eps
{ munger => 'eps', prefix => 'http-req:', from => 'src_ip' }
{ munger => 'eps', prefix => 'dns-nxd:', from => 'src_ip',
read => 'rate', mark => 0 }
# multi-output: one daemon round trip fills several columns
{ munger => 'eps', prefix => 'http-req:', from => 'src_ip',
parts => [ 'rate', 'count' ], into => [ 'req_rate', 'req_count' ] }Per-entity sliding-window event rates via the iqbi-damiq daemon shipped with Algorithm::EventsPerSecond (see Algorithm::EventsPerSecond::Sukkal). The input value becomes a meter key (after prefix is prepended); by default the munger marks one event against that key and returns the key's current events-per-second, using the daemon's MARKRATE command -- mark and query in a single command with a single reply. This is the munger behind rate columns like a per-source request rate: every event marks its source's meter and stores the rate the meter now reads.
Unlike every other munger this one consults external state -- but the state lives in the daemon, not here, so the munger itself remains a stateless client and rows stay reproducible given the daemon. Because the daemon is shared, multiple writer processes marking the same keys see one global rate, which an in-process meter could never give.
Spec keys:
socket- unix socket path of the daemon. Defaults to$Algorithm::ToNumberMunger::EPS_SOCKET(/var/run/iqbi-damiq.sock).prefix- string prepended to the input to form the key, namespacing this column's meters (two columns keyed on the same field need different prefixes or they share meters). No whitespace/control characters. Default''.mark- whether to mark the key (default1). Marking ridesMARKRATE: withread => 'rate'that one command is the whole exchange; withcount/totalthe read is pipelined after it (theMARKRATErate reply is discarded), so a marking failure still comes back as an ordinary first reply. Withmark => 0the munger only reads, for columns whose marking is done elsewhere -- e.g. an NXDOMAIN rate is marked by the pipeline only on NXDOMAIN responses but read on every row.read- what to read:rate(events/sec over the daemon's window, default),count(events inside the window), ortotal(lifetime).parts+into- multi-output form (see "compile"): read several ofrate/count/totalfor the one key in a single round trip, filling one column each. When marking, theMARKRATEreply itself serves the firstratepart, soparts => ['rate', 'count']costs exactly two commands.on_error-'die'(default) croaks the write when the daemon is unreachable or repliesERR; a number is returned instead as a quiet fallback. Note0is indistinguishable from a genuinely idle key, so quiet fallback biases the column -- loud is the default on purpose.timeout- per-operation socket timeout in seconds (default 5, best-effort viaSO_RCVTIMEO/SO_SNDTIMEO), so a wedged daemon cannot hang a writer forever.
Semantics worth knowing: a marked read includes the event just marked; keys have whitespace/control bytes replaced with _ to satisfy the daemon's key rules; connections are made lazily on first use and kept open (reconnecting transparently after a fork or an error), so compiling a plan -- including the eager validation in write_info -- needs no running daemon. Each eps column costs one unix-socket round trip per row; the multi-output form exists so rate+count of the same key costs one round trip, not two.
AUTHOR
Zane C. Bowers-Hadley, <vvelox at vvelox.net>
LICENSE AND COPYRIGHT
This software is Copyright (c) 2026 by Zane C. Bowers-Hadley.
This is free software, licensed under:
The GNU Lesser General Public License, Version 2.1, February 1999