NAME
Langertha::Holmgang::Report - The JSON report of a Holmgang invocation, format holmgang_report 1
VERSION
version 0.001
SYNOPSIS
use Langertha::Holmgang::Report;
my $report = Langertha::Holmgang::Report->new( with_transcript => 0 );
$report->add_run(
target => { label => 'local-a', engine => 'OpenAI', model => 'm-1', url => $url },
variant => { name => 'default', settings => {} },
suite => { id => 'demo', version => '1', topic => 'demo', digest => $digest },
repeat => 1, status => 'complete', error => undef,
cases => [ { case_id => 'demo-01', title => 'Open the port', tags => [], tier => undef,
difficulty => undef, weight => 1, hard => 0, untrusted => 0 } ],
results => [ $result_from_the_runner ],
);
$report->finish->write('/data/reports/run.json');
my $again = Langertha::Holmgang::Report->read('/data/reports/run.json');
my $all = Langertha::Holmgang::Report->merge( $again, $other );
DESCRIPTION
One JSON document per invocation in format holmgang_report: 1, exactly as ADR 0006 fixes it: canonical UTF-8 JSON, every field always present, null for an unknown value (never 0, never absent), booleans as true/false. It holds what was measured, per run, case and repetition — never a derived number beyond score, tokens and the call indices; topics, weights, curves and verdicts are computed from it elsewhere, so merging reports cannot count anything twice.
No credentials leave through it: the target carries only label, engine, model and a url stripped of user info and credential parameters; error texts must have passed the engine's scrubbing before they arrive.
generator
{ holmgang => $version, langertha => $version }, the versions that wrote the report. Defaults to the loaded ones. Volatile.
started_at
ISO-8601 time the invocation started. Defaults to the time the report object was first asked for it; pass undef for unknown. Volatile.
finished_at
ISO-8601 time the invocation finished, undef until "finish". Volatile.
with_transcript
The switch of ADR 0006 for transcript: when true, "add_run" keeps each result's transcript (without system messages, the untrusted mark stripped); off by default, and then every repetition's transcript is null.
runs
The runs in format 1, as "add_run" builds them. Runs passed to new must already be in format 1; they are checked and normalised the same way.
now
my $ts = $report->now; # 2026-11-04T21:13:52Z
my $ts = $report->now($epoch);
The ISO-8601 UTC time format the report uses.
add_run
$report->add_run(
target => { label => 'local-a', engine => 'OpenAI', model => 'm-1',
url => 'http://10.0.0.5:8000/v1' },
variant => { name => 'think-off', settings => { enable_thinking => 0 } },
suite => { id => 'demo', version => '3', topic => 'demo', digest => '…' },
repeat => 3,
status => 'complete', # complete | cancelled | aborted
error => undef, # run error, already scrubbed by the engine
cases => [ <case meta>, … ], # in the suite's load order
results => [ <result>, … ], # one per case and repetition, any order
);
Builds one run of format 1 from the runner's results and appends it; returns the run hash. Only the fields of format 1 are copied, so anything else a caller passes (a key in target, the case state) never reaches the report. url loses its user info, fragment and every query parameter whose name looks like a credential. Croaks on a shape error (with the field path), a result for a case not in cases, a repetition given twice or beyond repeat, an object or image content anywhere (see "IMAGES").
Case meta (the case's own fields, ADR 0005): case_id (required), title, tags, tier, difficulty, weight, hard, untrusted. A case without results gets repetitions => [].
Result — the per-repetition interface the runner produces, plain data:
{
case_id => 'demo-01',
repetition => 1, # 1..repeat
seed => 1, # int or undef
status => 'passed', # wire name, ADR 0006
score => 0.75, # 0..1 from the grader
error => undef, # scrubbed text or undef
rules => [ { title, kind, required, passed, reason, sim_gap }, … ],
diff => [ { path, before, after }, … ], # path as text
turns => [ { # one per model call that reached the target
input_tokens, output_tokens, reasoning_tokens, # undef when not sent
duration_ms, finish_reason, calls, # calls = tool calls of the turn
}, … ],
calls => [ { tool, args, succeeded, sim_gap }, … ], # in call order
forbidden_calls => 0,
injection_held => undef, # undef without untrusted content
transcript => [ … ], # messages; kept only with with_transcript
}
Booleans may be any Perl truth value; they are written as JSON true/false. A missing key is null in the report. The report derives, and the result does not carry: duration_ms (sum of the turns' duration_ms, 0 without turns), finish_reason (of the last turn), tokens (per field the sum over the turns, null when no turn had a value; input_max the largest input_tokens) and sim_gaps (one entry per call with a sim_gap, call its 0-based index). A turn that never reached the target (the no_vision pre-check) is not in turns. Transcript images must already be "image_entry" placeholders.
finish
$report->finish; # finished_at = now
$report->finish($iso_time);
image_entry
my $placeholder = $report->image_entry( 'scan-1.png', $raw_bytes );
# { image => 'scan-1.png', sha256 => '…', bytes => 48213 }
The only form an image takes in a report (ADR 0006). Also callable on the class.
as_data
The report as a plain hash of format 1.
to_json
The report as canonical UTF-8 JSON bytes. The same input gives the same bytes; the volatile fields (generator, started_at, finished_at, each repetition's duration_ms and each turn's duration_ms) are the only ones that differ between two runs of the same mock.
write
$report->write('/data/reports/run.json');
Writes "to_json" to the file. Reports of private suites are as private as their cases (ADR 0003): write them outside the checkout.
from_json
my $report = Langertha::Holmgang::Report->from_json( $bytes, source => $name );
Reads a report. Croaks loudly on anything but JSON, on a missing or higher holmgang_report version, and on a shape error with the field path: every field of format 1 must be present (null where unknown) — except difficulty of a case, added to format 1 later (k57), which reads as null where a report lacks it —, booleans must be JSON booleans, image content is refused. Fields it does not know are ignored and not kept.
read
my $report = Langertha::Holmgang::Report->read('/data/reports/run.json');
merge
my $all = Langertha::Holmgang::Report->merge( map { Langertha::Holmgang::Report->read($_) } @files );
One report from several, without counting anything twice: runs with the same target, variant and suite become one, their cases are joined by case_id and the repetitions by number. A repetition found in more than one report is kept once when it is identical apart from the volatile fields, and croaks when it differs; so does a different repeat or case meta for the same run, except that a difficulty of null (a report written before the field came, k57) takes the other report's value. The merged run's status is the worst of its parts (aborted over cancelled over complete), error the first one given; started_at is the earliest, finished_at the latest (compared as text, as this module writes them in UTC), generator that of the first report.
from_cai_platform
my $report = Langertha::Holmgang::Report->from_cai_platform(
$decoded_cai_report,
grading => { <cai run id> => { <case_id> => <grading JSON from the database> } },
suites => { <suite id> => { version => …, digest => …,
cases => { <case_id> => { tags, tier, difficulty, weight, hard, untrusted } } } },
);
Maps a cai-platform evals run --report file onto format 1 for the acceptance comparison, following the table in ADR 0006. Rows with the same target and suite, in their order in runs (start order), become repetitions 1..n of one run; repeat comes from repeat_by_suite. Statuses: passed stays, failed/ungraded with exceeded become turn_limit, ungraded becomes uncertain, failed, error and context_exceeded keep their names; an unknown status croaks. Run statuses done, cancelled, failed become complete, cancelled, aborted.
The --report file holds counts, not lists. So turns and calls get as many entries as the counts say, with null fields; token sums of 0 become null; sim_gaps keeps only the gap texts. Rules and the diff are only in the database: with grading for a case they are mapped (detail to reason, 0/1 to booleans, the rule of kind turns dropped and score recomputed as in Eval::Grader), the gaps get their call index and tool, and the calls their tool and gap. Without grading, rules and diff are null, and score is null for a turn limit (it counted the dropped rule). Case meta beyond case_id, title and difficulty (each case row of --report carries it; suites fills in where a row lacks it), the suite version and digest come from suites or are null; target.engine and the variant are always null; seed is the round number when repeat is above 1, as Eval::Batch sets it. Timestamps are null.
IMAGES
No image content goes into a report. Anywhere in the document, a data: URL, a base64 run of 1024 characters or more, a provider image block carrying base64 (source => { type => 'base64' }, inline_data, inlineData, b64_json) or any object (such as a Langertha::Content::Image) is refused on write and on read. An image in a transcript is the placeholder "image_entry" builds.
SEE ALSO
SUPPORT
Issues
Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.
IRC
Join #langertha on irc.perl.org or message Getty directly.
CONTRIBUTING
Contributions are welcome! Please fork the repository and submit a pull request.
AUTHOR
Torsten Raudssus <torsten@raudssus.de>
COPYRIGHT AND LICENSE
This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.