NAME
Langertha::Holmgang::Batch - Suites against targets and their variants, repeated, with bounded concurrent requests per target, resumable
VERSION
version 0.001
SYNOPSIS
use Langertha::Holmgang::Batch;
my $batch = Langertha::Holmgang::Batch->new(
suites => [ $loader->load($dir_a), $loader->load($dir_b) ],
targets => [ {
target => { label => 'local-a', engine => 'vLLM', model => 'm-1', url => 'http://10.0.0.5:8000/v1' },
engine => sub { my ( $case, $slot ) = @_; $engines[$slot] //= build_engine() },
variants => [
{ name => 'think-off', settings => { enable_thinking => 'off' } },
{ name => 'think-on', settings => { enable_thinking => 'on', max_tokens => 16000, deadline => 1800 } },
],
concurrency => 4, # at most 4 requests at once to this target
} ],
repeat => 3, # else each suite's own repeat, else 1
resume => Langertha::Holmgang::Report->read('out.json'), # optional
on_event => sub { my ( $event ) = @_; print "$event->{type}\n" },
);
my $report = $batch->run; # or: await $batch->run_f
$batch->cancel; # from a signal handler: no new case starts
DESCRIPTION
The run over suites (cai-platform Eval::Batch::run and Eval::Runner::run): for every target, every variant of it and every suite, one run (ADR 0006) with its repeat repetitions of every case, each case played by Langertha::Holmgang::Runner. The result is a Langertha::Holmgang::Report with one run per target, variant and suite, in that order.
Concurrency. Each target has its own bound (concurrency, default 1): at most that many cases of that target are in play at once, so at most that many requests are open against it. Different targets run side by side. A case in play holds a slot (0 .. concurrency-1) and gets its engine from the target's engine code with that slot, so a caller can give every slot its own engine (and with it its own connection). Within a target the cases start in this order: variant, suite, repetition, case. With concurrency 1 that is one request at a time, as cai-platform's batch ran.
Variants. A variant is a name and its settings, named and checked by "check_variant" in Langertha::Holmgang::Target (enable_thinking, reasoning_effort, max_tokens, temperature, deadline), against the engine when the batch is built. The report carries the name and the normalised settings.
Vision. Before the cases of a run whose suite has images, the target's "vision_check_f" in Langertha::Holmgang::Target runs once (with the run's variant); its no_vision goes to every case of the run.
Resume. With resume, an earlier report of the same batch: every repetition it holds a result for (whatever the status) is taken over and not played again; only the missing ones run. A run of the earlier report must match one of the batch (target, variant, suite) with the same suite digest and the same repeat; a run the batch does not cover is refused, so resuming never drops results.
Errors. A target that fails a turn is a status of the case (error, context_exceeded, no_vision), not a batch error. A case that throws (a program error) aborts its run: no further case of that run starts, the run is aborted, its error the text through "scrub" in Langertha::Holmgang::Target (with the engine's key) and cut to "error_max_length" characters; the other runs go on. Each repetition's error is cut to the same length.
Cancel. "cancel" starts no further case; cases in play finish. A run that then misses results is cancelled.
suites
Required. The Langertha::Holmgang::Suites, loaded (and so linted), in the order of the report.
targets
Required. One HashRef per target:
engine-- required: a code reference( $case, $slot )that returns the engine to play the case on, or one engine for every case.target-- the report's target,{ label, engine, model, url }(ADR 0006); never a key.variants--[ { name, settings }, ... ]; default one variantdefaultwithout settings. Names are unique per target.concurrency-- the bound, a whole number from 1; default 1.
repeat
Optional. Repetitions for every suite (1 to 100); without it each suite's own repeat, else 1.
resume
Optional. An earlier Langertha::Holmgang::Report of the same batch (see "DESCRIPTION").
on_event
Optional. A code reference called with a HashRef as things happen, in the order they happen. Every event carries type, target (the report's target), variant (its name) and suite (the suite object):
result-- a case played:case,repetition,result(the runner's result, "run_case_f" in Langertha::Holmgang::Runner);vision-- the vision pre-check of a run:check(the HashRef of "vision_check_f" in Langertha::Holmgang::Target);aborted-- a run aborted:error(scrubbed), andcaseandrepetitionwhen a case threw, none when the vision check did.
An on_event that dies aborts the run the event belongs to.
error_max_length
Langertha::Holmgang::Batch->error_max_length; # 2000
How many characters of an error a run and a repetition keep (cai-platform cuts at the same length when it stores them).
cancel
$batch->cancel;
No further case starts; the cases in play finish. Safe to call from a signal handler and from "on_event".
cancelled
True once "cancel" was called.
run_f
my $report = await $batch->run_f;
Runs the batch; a Future of the final "report". Runs once.
run
my $report = $batch->run;
"run_f", waited for; runs the IO::Async::Loop while requests are open.
report
my $report = $batch->report;
The Langertha::Holmgang::Report of the batch as it stands: one run per target, variant and suite, the results taken over from "resume" and those played so far. Callable at any time, so a caller can write a checkpoint after each result: a run still in play is aborted there with the error unfinished: ..., which a later "resume" continues. After "run" it is the final report, with finished_at; with "resume" its started_at is the earlier report's.
SEE ALSO
SUPPORT
Issues
Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.
IRC
Join #langertha on irc.perl.org or message Getty directly.
CONTRIBUTING
Contributions are welcome! Please fork the repository and submit a pull request.
AUTHOR
Torsten Raudssus <torsten@raudssus.de>
COPYRIGHT AND LICENSE
This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.