NAME
Langertha::Holmgang::Runner - The model loop of one case over a Langertha engine: turns, turn limit, tokens per turn, lint mock runs
VERSION
version 0.001
SYNOPSIS
use Langertha::Holmgang::Runner;
use Langertha::Holmgang::Target;
use Langertha::Engine::vLLM;
# any Langertha engine; the key, where one is needed, comes from the engine's environment variable
my $engine = Langertha::Engine::vLLM->new( url => 'http://10.0.0.5:8000/v1', model => 'some-model' );
# or, without a network: the mock model that replays the case's reference solution
my $engine = Langertha::Holmgang::Runner->mock_engine( $case, 'solution' );
my $runner = Langertha::Holmgang::Runner->new(
target => Langertha::Holmgang::Target->new( engine => $engine, variant => { max_tokens => 4000 } ),
);
my $result = $runner->run_case( $case, repetition => 1, seed => 1 );
$result->{status}; # passed failed uncertain turn_limit error context_exceeded no_vision
$result->{turns}; # [ { input_tokens, output_tokens, reasoning_tokens, duration_ms, finish_reason, calls } ]
my $result = await $runner->run_case_f($case); # the same, as a Future
# the results go straight into a report (ADR 0006)
$report->add_run( %run, cases => \@case_meta, results => [ $result, … ] );
# lint's two mock runs, for Langertha::Holmgang::Lint->new( mock_runs => … )
my $lint = Langertha::Holmgang::Lint->new(
mock_runs => Langertha::Holmgang::Runner->lint_mock_runs,
);
DESCRIPTION
The runner plays one case against a model (cai-platform Eval::Runner::run_case, here over Langertha instead of ModelTester). Per case: a fresh Langertha::Holmgang::World from the case's initial state, then turns against the "target". A turn sends the system prompt, the conversation and the suite's tool schemas through "chat_turn_f" in Langertha::Holmgang::Target; every tool call of the answer runs in the world and its output goes back as a tool result in the engine's own wire format ("format_tool_results" in Langertha::Role::Tools), so any engine with native or Hermes tool calling works. The case ends when the model answers without a tool call, after max_turns model calls (status turn_limit), or when the target does not answer (error, context_exceeded or no_vision, as the target classifies the failure). Only the end state and the calls are graded (Langertha::Holmgang::Grader), never the text.
What goes out is the system prompt (the case's effective system), the case's "prompt_messages" in Langertha::Holmgang::Case (role and content only, no untrusted mark; each image placeholder replaced by the image itself, "image_content" in Langertha::Holmgang::Case, only in the request -- the transcript keeps the placeholder), the assistant turns and tool outputs, and the tool schemas as "offer" in Langertha::Holmgang::World gives them (name, description, parameters). Never the state, the observations, the rules, the solutions or the marks. An engine with a system_prompt of its own is refused before the first request: Langertha would send it as a second system message ahead of the case's, and the model would see what the case does not say.
Request controls per turn: max_tokens (effective value of the case), seed when given, and temperature (effective value) only when the engine takes one (supports('temperature')) -- Langertha would drop it with a warning otherwise. The target's variant settings win over both ("variant" in Langertha::Holmgang::Target).
target
The Langertha::Holmgang::Target the turns go to. Required for "run_case"; "lint_mock_runs" builds its own.
grader
The Langertha::Holmgang::Grader that grades the end state; its simulator registry also runs the world's calls.
run_case_f
my $result = await $runner->run_case_f( $case, repetition => 2, seed => 2 );
Plays one Langertha::Holmgang::Case (see "DESCRIPTION"). seed (integer, optional) goes unchanged into every request of the case; repetition (default 1) is only carried into the result. no_vision (optional) is the reason the vision pre-check of the suite gave ("vision_check_f" in Langertha::Holmgang::Target): a case with images then ends no_vision with that text as its error, without a request and without a turn; a case without images ignores it. Returns a Future of the result, the per-repetition shape "add_run" in Langertha::Holmgang::Report takes (ADR 0006), plain data:
case_id,repetition,seedstatus,score,error,rules,diff,sim_gaps,forbidden_calls,injection_held-- from "grade" in Langertha::Holmgang::Grader.erroris the target's error text when it did not answer, scrubbed ("scrub" in Langertha::Holmgang::Target),undefotherwise (a turn limit has none). A failed request ends the case: its text never goes back to the model.turns-- one entry per model call that reached the target, the failed one included:input_tokens,output_tokens,reasoning_tokens(from Langertha::Usage,undefwhen the target sent none),duration_ms(wall time of the call),finish_reason,calls(tool calls of the turn).calls-- every tool call in order:tool,args(decoded),succeeded,sim_gap.transcript-- the conversation without the system prompt, in the engine's wire format, ending with the final assistant text.answer-- the last text of the model (raw, only to look at), andstate-- the end state. Neither goes into a report.
run_case
my $result = $runner->run_case( $case, %options );
"run_case_f", waited for.
mock_engine
my $engine = Langertha::Holmgang::Runner->mock_engine( $case, 'solution' ); # or 'text'
A Langertha::Engine::Virtual as the mock model of the lint (and of dry runs): solution calls the reference solution, one call per turn, then answers Done.; text answers with text only, at once. For a case with images it claims image_input (Langertha::Role::ImageInput), answers OK once its script is done (which a case never reaches) and reports no token usage, so "vision_check_f" in Langertha::Holmgang::Target lets it through unmeasured. It replaces cai-platform's Eval::FakeTarget (ADR 0005); unlike that one it estimates no tokens and emulates no context window (the target's error path is tested with "error" in Langertha::Virtual::Rules reactions instead).
lint_mock_runs
my $lint = Langertha::Holmgang::Lint->new( mock_runs => Langertha::Holmgang::Runner->lint_mock_runs );
The code reference for "mock_runs" in Langertha::Holmgang::Lint: the two mock runs of the lint (cai-platform docs/entwurf-evals.md §3.5), each a real "run_case" over a "mock_engine", graded with the lint's grader.
Replaying the reference solution passes, twice identically -- status, score, rules, diff, calls, gaps, transcript, end state and the turns without their wall time.
Answering text only fails -- except in the "no tool" family (
solution: []), where it must pass.
A run that does not hold throws a Langertha::Holmgang::Error on the field solution or rules; the case does not load.
SUPPORT
Issues
Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.
IRC
Join #langertha on irc.perl.org or message Getty directly.
CONTRIBUTING
Contributions are welcome! Please fork the repository and submit a pull request.
AUTHOR
Torsten Raudssus <torsten@raudssus.de>
COPYRIGHT AND LICENSE
This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.