NAME

Langertha::Holmgang::Batch - Suites against targets and their variants, repeated, with bounded concurrent requests per target, resumable

VERSION

version 0.001

SYNOPSIS

use Langertha::Holmgang::Batch;

my $batch = Langertha::Holmgang::Batch->new(
  suites  => [ $loader->load($dir_a), $loader->load($dir_b) ],
  targets => [ {
    target      => { label => 'local-a', engine => 'vLLM', model => 'm-1', url => 'http://10.0.0.5:8000/v1' },
    engine      => sub { my ( $case, $slot ) = @_; $engines[$slot] //= build_engine() },
    variants    => [
      { name => 'think-off', settings => { enable_thinking => 'off' } },
      { name => 'think-on',  settings => { enable_thinking => 'on', max_tokens => 16000, deadline => 1800 } },
    ],
    concurrency => 4,                          # at most 4 requests at once to this target
  } ],
  repeat   => 3,                               # else each suite's own repeat, else 1
  resume   => Langertha::Holmgang::Report->read('out.json'),   # optional
  on_event => sub { my ( $event ) = @_; print "$event->{type}\n" },
);

my $report = $batch->run;                      # or: await $batch->run_f
$batch->cancel;                                # from a signal handler: no new case starts

DESCRIPTION

The run over suites (cai-platform Eval::Batch::run and Eval::Runner::run): for every target, every variant of it and every suite, one run (ADR 0006) with its repeat repetitions of every case, each case played by Langertha::Holmgang::Runner. The result is a Langertha::Holmgang::Report with one run per target, variant and suite, in that order.

Concurrency. Each target has its own bound (concurrency, default 1): at most that many cases of that target are in play at once, so at most that many requests are open against it. Different targets run side by side. A case in play holds a slot (0 .. concurrency-1) and gets its engine from the target's engine code with that slot, so a caller can give every slot its own engine (and with it its own connection). Within a target the cases start in this order: variant, suite, repetition, case. With concurrency 1 that is one request at a time, as cai-platform's batch ran.

Variants. A variant is a name and its settings, named and checked by "check_variant" in Langertha::Holmgang::Target (enable_thinking, reasoning_effort, max_tokens, temperature, deadline), against the engine when the batch is built. The report carries the name and the normalised settings.

Vision. Before the cases of a run whose suite has images, the target's "vision_check_f" in Langertha::Holmgang::Target runs once (with the run's variant); its no_vision goes to every case of the run.

Resume. With resume, an earlier report of the same batch: every repetition it holds a result for (whatever the status) is taken over and not played again; only the missing ones run. A run of the earlier report must match one of the batch (target, variant, suite) with the same suite digest and the same repeat; a run the batch does not cover is refused, so resuming never drops results.

Errors. A target that fails a turn is a status of the case (error, context_exceeded, no_vision), not a batch error. A case that throws (a program error) aborts its run: no further case of that run starts, the run is aborted, its error the text through "scrub" in Langertha::Holmgang::Target (with the engine's key) and cut to "error_max_length" characters; the other runs go on. Each repetition's error is cut to the same length.

Cancel. "cancel" starts no further case; cases in play finish. A run that then misses results is cancelled.

suites

Required. The Langertha::Holmgang::Suites, loaded (and so linted), in the order of the report.

targets

Required. One HashRef per target:

  • engine -- required: a code reference ( $case, $slot ) that returns the engine to play the case on, or one engine for every case.

  • target -- the report's target, { label, engine, model, url } (ADR 0006); never a key.

  • variants -- [ { name, settings }, ... ]; default one variant default without settings. Names are unique per target.

  • concurrency -- the bound, a whole number from 1; default 1.

repeat

Optional. Repetitions for every suite (1 to 100); without it each suite's own repeat, else 1.

resume

Optional. An earlier Langertha::Holmgang::Report of the same batch (see "DESCRIPTION").

on_event

Optional. A code reference called with a HashRef as things happen, in the order they happen. Every event carries type, target (the report's target), variant (its name) and suite (the suite object):

An on_event that dies aborts the run the event belongs to.

error_max_length

Langertha::Holmgang::Batch->error_max_length;   # 2000

How many characters of an error a run and a repetition keep (cai-platform cuts at the same length when it stores them).

cancel

$batch->cancel;

No further case starts; the cases in play finish. Safe to call from a signal handler and from "on_event".

cancelled

True once "cancel" was called.

run_f

my $report = await $batch->run_f;

Runs the batch; a Future of the final "report". Runs once.

run

my $report = $batch->run;

"run_f", waited for; runs the IO::Async::Loop while requests are open.

report

my $report = $batch->report;

The Langertha::Holmgang::Report of the batch as it stands: one run per target, variant and suite, the results taken over from "resume" and those played so far. Callable at any time, so a caller can write a checkpoint after each result: a run still in play is aborted there with the error unfinished: ..., which a later "resume" continues. After "run" it is the final report, with finished_at; with "resume" its started_at is the earlier report's.

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.