NAME

Langertha::Holmgang::Haystack - Long, deterministic operational data for long-context cases, generated from a short spec

VERSION

version 0.001

SYNOPSIS

use Langertha::Holmgang::Haystack;

my $haystack = Langertha::Holmgang::Haystack->new;

my $log = $haystack->generate( {
  kind    => 'journal',            # journal syslog nginx-access postgres-log ini-config k8s-manifests runbooks
  tokens  => 64000,                # size of the filler (estimated)
  seed    => 7,
  host    => 'app7',
  units   => [qw( belegimport lieferschein )],
  exclude => [ 'Zu viele offene Dateien' ],   # never occurs in the filler
  insert  => [
    { at => 0.73, text => '{ts} {host} belegimport[{pid:belegimport}]: FEHLER: Zu viele offene Dateien' },
    { from => 0.5, to => 0.625, count => 6, text => '...' },
  ],
} );

# in the loader: a message or file content made of parts
my $text = $haystack->resolve(
  [ "Hier der Auszug:\n", { generate => { ... } }, "\nBitte ..." ],
  'cases/x.yaml: messages[0].content',
);

$haystack->estimate_tokens($text);                          # characters / 3.5
Langertha::Holmgang::Haystack->estimate_tokens($text);      # same, as a class method

DESCRIPTION

Long-context cases need excerpts of 16k to 160k tokens; as YAML in a suite that would be megabytes. Instead a case carries a short spec { generate => { ... } } in a message content or in a file content of the state, and this module turns it into text.

Deterministic: the same spec gives the same text, byte for byte, on every perl with 64-bit integers and independent of PERL_HASH_SEED (its own xorshift32 random source, never rand). The algorithm and its vocabulary are those of cai-platform's CAI::Platform::Eval::Haystack, and the text is the same as there, so a case measures the same input on both. When the generator changes, the text changes — the loader therefore puts the generated text into the suite digest.

Content: plausible German operational data — messages of invented services in German, comments in configurations in German, plus the usual English lines of systemd, cron and sshd. Host and service names are invented, addresses come from 10.20.0.0/16 and the documentation networks, names lie under example.invalid. Time runs evenly from start over hours hours (line position maps to time of day): an insert at at: 0.5 with start 22:00 and 8 hours stands at 02:xx.

Size: estimated with "chars_per_token" (3.5 characters per token, an assumption for German log text with numbers and timestamps under common BPE tokenizers). tokens is the size of the filler; inserts come on top (they are small), so the filler does not depend on them.

The filler gives nothing away: every line (or block) of the filler that a pattern from exclude matches is dropped and drawn again; at the end the whole filler text (with timestamps) is checked once more. With inserts, every pattern must match at least one of them — so the needle is exactly where the case puts it.

Results are cached per instance.

THE SPEC

kind:      journal | syslog | nginx-access | postgres-log | ini-config
           | k8s-manifests | runbooks   (required)
tokens:    1000 .. 200000  (required; size of the filler, estimated)
seed:      integer 0 .. 2**31 (default 1)
start:     "2026-10-28 22:00:00" (local time, default)   hours: 0.5 .. 72 (default 8)
tz:        CET | CEST (default CET)
host:      host of the journal (default srv1)
hosts:     hosts of a central syslog (default [web1, web2, web3])
units:     services in journal/syslog (default: six invented ones)
upstreams: upstream addresses in the nginx log (default 10.20.2.11..14:8080)
database:  database in the PostgreSQL log (default shop)
title:     name in the header (ini-config: configuration, k8s-manifests:
           release, runbooks: handbook)
namespace: namespace of the manifests (k8s-manifests, default shop)
exclude:   [ regular expression, ... ]  (multi-line, ^/$ per line)
insert:    [ { at: 0..1, text }, { from, to, count, text }, ... ]

Placeholders in text: {ts} (timestamp in the kind's format at that position), {host}, {pid:service} or {pid:host/service} (the same PID as in the filler), {num}, {ip}, {ms}, {port} (random values). from/to/count spreads count inserts evenly over the range. In configurations (blocks instead of lines) each insert is a block of its own between two sections.

Blocks: k8s-manifests is a rendered bundle of Kubernetes manifests (YAML documents separated by ---): per application a Deployment and a Service, sometimes a ConfigMap — consistent in the filler (selector = labels, targetPort hits a containerPort or its name), names unique. runbooks is an export of operations runbooks (Markdown) with title, Stand: date, occasion and steps for invented services. Inserts are whole documents or sections.

chars_per_token

Characters per token for "estimate_tokens" and the size of the filler. Default 3.5. A different value gives a different text for the same spec.

kinds

my @kinds = Langertha::Holmgang::Haystack->kinds;

The known kinds: journal, syslog, nginx-access, postgres-log, ini-config, k8s-manifests, runbooks.

estimate_tokens

my $n = $haystack->estimate_tokens($text);
my $n = Langertha::Holmgang::Haystack->estimate_tokens($text);

Estimated tokens of a text: characters divided by "chars_per_token", rounded. As a class method it uses the default 3.5. Undefined text is 0.

generate

my $text = $haystack->generate( \%spec, $field );

Returns the text for the spec (see "THE SPEC"): a character string ending in a newline. $field (default generate) is the field path a refusal starts with. Throws a Langertha::Holmgang::Error (as a string <field>: <reason>) when the spec does not hold — unknown key, size outside 1000..200000, an insert without exclude, a pattern that matches no insert, one that matches the filler (its timestamps, say), a {pid:...} of a service that does not exist. Cached per instance.

resolve

my $text = $haystack->resolve( $value, $field );

Resolves a content as the loader finds it: a hash { generate => \%spec } becomes the generated text ("generate"), a list of parts (text or such a hash) the joined text. Any other value comes back unchanged. Throws a Langertha::Holmgang::Error (as a string <field>: <reason>); the field of a spec is <field>.generate, of a part <field>[<i>].

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.