NAME

Langertha::Holmgang::Evaluation - Per-topic evaluation of Holmgang reports: weights, length tiers, verdicts, comparison

VERSION

version 0.001

SYNOPSIS

use Langertha::Holmgang::Evaluation;
use Langertha::Holmgang::Report;

my $evaluation = Langertha::Holmgang::Evaluation->new(
  reports    => [ map { Langertha::Holmgang::Report->read($_) } @files ],
  thresholds => { 'demo-suite' => { pass_rate => 0.9, pass_k => 0.5 } },
);
print $evaluation->to_text;
my $ok = !grep { !$_->{ok} } @{ $evaluation->verdicts };

DESCRIPTION

The numbers derived from one or more Langertha::Holmgang::Reports, which hold only what was measured (ADR 0006): per topic, never one share over all cases; an overall value only as the mean of the topics; case weights (cluster size / representatives); curves per length tier with hits, context_exceeded and errors apart, and the same per difficulty; the verdict per suite against its threshold and its hard cases, with pass^k over the repetitions; the comparison of targets as text, Markdown and JSON. Port of cai-platform's Eval::Batch::by_topic and Eval::Batch::verdicts; the decisions are ADR 0008.

Rates. Every rate is a weighted share over all results, whatever their status: a result that was not measured (no_vision, context_exceeded, error) is in the denominator and has not passed. The counters say why a rate is low — each of the seven statuses is counted under its own name.

reports

Required. The Langertha::Holmgang::Report objects to evaluate, as an array reference; report => $report is the same for one. They are joined with "merge" in Langertha::Holmgang::Report first, so a repetition found in two files counts once and a conflict croaks.

thresholds

Optional. Per suite id the suite's threshold as the loader reads it from suite.yaml: { 'suite-id' => { pass_rate => 0.9, pass_k => 0.5 } }. A report does not carry the threshold, so a suite missing here is judged without one: by its hard cases, an unfinished run and whether anything was measured ("verdicts").

topics

# [ { target, variant, topic, suites => [ ids ], results, weight,
#     counts => { <status> => n, … }, rate,
#     tokens => { input, output, reasoning } }, … ]

Per row (target and variant, in the order they first appear) and topic (by name): results is the number of repetitions with a result, counts counts each of the seven statuses under its own name, weight the sum of the case weights over those results, rate the weighted share of passed among all results (null without results). Tokens are sums, null when no repetition had a value.

overall

# [ { target, variant, topics, results, counts => { … }, rate }, … ]

Per row the mean of its topic rates — never one share over all cases, so a few hard cases do not drown in many easy ones. Topics without a rate do not enter the mean; rate is null when none has one. results and counts are plain sums over the topics.

tier_curves

# [ { target, variant, topic, tier, results, weight, counts => { … },
#     rate, context_exceeded_rate, error_rate }, … ]

The curve per length tier, per row and topic: tiers in the order 16k, 48k, 96k, 160k, any other tier after them by its number. Hits (rate), context_exceeded and error are three weighted shares of the same results, kept apart; counts has all seven statuses. Cases without a tier are not on a curve. The tag ctx-<N>k is no tier (ADR 0008, k56).

difficulty_curves

# [ { target, variant, topic, difficulty, results, weight, counts => { … },
#     rate, context_exceeded_rate, error_rate }, … ]

The same curve per difficulty (k57): per row and topic, in the order leicht, mittel, schwer, any other value after them. Cases without a difficulty (and reports written before the field) are not on a curve.

verdicts

# [ { target, variant, suite_id, results, counts => { … }, passed, rate,
#     pass_rate, k, pass_k, pass_any, pass_k_min,
#     case_results => [ { case_id, passed, repetitions } ],
#     hard_failed  => [ { case_id, forbidden_calls, repetitions } ],
#     repetitions_error, unfinished => { status, error } | undef, ok }, … ]

The verdict per row and suite over all repetitions, in the order the runs first appear. rate is the weighted share of passed among all results; pass_rate and pass_k_min come from "thresholds".

Not ok when rate is below pass_rate (or there is no result), when pass_k is below pass_k_min (or there is no case), or when a hard case (tag injection or verbot) had a forbidden call in any repetition — whatever the rate. Never ok either, with or without a threshold, when unfinished is set: the run's status and error when its status is not complete (cancelled, aborted), so its results are a partial state even if all of them passed. Nor is it ok, with or without a threshold, when the suite measured nothing: repetitions_error equals k. Without a threshold the hard cases, unfinished and that decide.

repetitions_error is the number of repetitions (of k) in which no case has a result other than error, none at all included: the target was not reached (401, 429, timeout, connection refused) and nothing was measured. context_exceeded, no_vision and every other status count as reached, so a suite all context_exceeded is a measurement and can be ok. When every repetition measured nothing the verdict is not ok; when some did and at least one was measured, it is reported and noted in the text but does not decide ok (ADR 0008, k73 amendment; cai-platform's runs_error decides for any count). ok false with repetitions_error equal to k is what the text shows as NOT MEASURED.

Reliability: k is the run's repeat; pass_k is the weighted share of cases passed in all k repetitions, pass_any (pass@k) of those passed in at least one — over the cases with at least one result, null without any. Only passed counts as passed: a repetition that is missing (a cancelled run) or ended error, context_exceeded, no_vision, turn_limit or uncertain has not passed. case_results lists per case in how many repetitions it passed and how many it has.

comparison

# { topics => [ names ], rows => [ { target, variant, rates => { topic => rate }, overall } ] }

The comparison table: a row per target and variant, a column per topic, the cell the topic's rate, and overall the mean of the row's topics. A topic a row did not run is absent from its rates.

as_data

The whole evaluation as a plain hash: holmgang_evaluation (the format version, ADR 0008), "topics", "overall", "tier_curves", "difficulty_curves", "verdicts", "comparison".

to_json

"as_data" as canonical UTF-8 JSON bytes; the same reports give the same bytes.

to_text

print $evaluation->to_text;

The comparison table, the verdicts (with a note for an unfinished run, for repetitions without a measurement and for each hard case that failed one), when any case has a tier the tier curves, and when any case has a difficulty the difficulty curves — as aligned plain-text tables. Rates as percentages, - for none. A verdict reads ok, NOT MEASURED when the suite measured nothing (whatever else failed it; the notes say the rest), or NOT PASSED.

to_markdown

The same tables as "to_text", as Markdown pipe tables under ### headings; hard-case notes follow the verdict table as a list, each naming its target, variant and suite.

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.