NAME

Langertha::Holmgang::Runner - The model loop of one case over a Langertha engine: turns, turn limit, tokens per turn, lint mock runs

VERSION

version 0.001

SYNOPSIS

use Langertha::Holmgang::Runner;
use Langertha::Holmgang::Target;
use Langertha::Engine::vLLM;

# any Langertha engine; the key, where one is needed, comes from the engine's environment variable
my $engine = Langertha::Engine::vLLM->new( url => 'http://10.0.0.5:8000/v1', model => 'some-model' );

# or, without a network: the mock model that replays the case's reference solution
my $engine = Langertha::Holmgang::Runner->mock_engine( $case, 'solution' );

my $runner = Langertha::Holmgang::Runner->new(
  target => Langertha::Holmgang::Target->new( engine => $engine, variant => { max_tokens => 4000 } ),
);

my $result = $runner->run_case( $case, repetition => 1, seed => 1 );
$result->{status};          # passed failed uncertain turn_limit error context_exceeded no_vision
$result->{turns};           # [ { input_tokens, output_tokens, reasoning_tokens, duration_ms, finish_reason, calls } ]

my $result = await $runner->run_case_f($case);    # the same, as a Future

# the results go straight into a report (ADR 0006)
$report->add_run( %run, cases => \@case_meta, results => [ $result, … ] );

# lint's two mock runs, for Langertha::Holmgang::Lint->new( mock_runs => … )
my $lint = Langertha::Holmgang::Lint->new(
  mock_runs => Langertha::Holmgang::Runner->lint_mock_runs,
);

DESCRIPTION

The runner plays one case against a model (cai-platform Eval::Runner::run_case, here over Langertha instead of ModelTester). Per case: a fresh Langertha::Holmgang::World from the case's initial state, then turns against the "target". A turn sends the system prompt, the conversation and the suite's tool schemas through "chat_turn_f" in Langertha::Holmgang::Target; every tool call of the answer runs in the world and its output goes back as a tool result in the engine's own wire format ("format_tool_results" in Langertha::Role::Tools), so any engine with native or Hermes tool calling works. The case ends when the model answers without a tool call, after max_turns model calls (status turn_limit), or when the target does not answer (error, context_exceeded or no_vision, as the target classifies the failure). Only the end state and the calls are graded (Langertha::Holmgang::Grader), never the text.

What goes out is the system prompt (the case's effective system), the case's "prompt_messages" in Langertha::Holmgang::Case (role and content only, no untrusted mark; each image placeholder replaced by the image itself, "image_content" in Langertha::Holmgang::Case, only in the request -- the transcript keeps the placeholder), the assistant turns and tool outputs, and the tool schemas as "offer" in Langertha::Holmgang::World gives them (name, description, parameters). Never the state, the observations, the rules, the solutions or the marks. An engine with a system_prompt of its own is refused before the first request: Langertha would send it as a second system message ahead of the case's, and the model would see what the case does not say.

Request controls per turn: max_tokens (effective value of the case), seed when given, and temperature (effective value) only when the engine takes one (supports('temperature')) -- Langertha would drop it with a warning otherwise. The target's variant settings win over both ("variant" in Langertha::Holmgang::Target).

target

The Langertha::Holmgang::Target the turns go to. Required for "run_case"; "lint_mock_runs" builds its own.

grader

The Langertha::Holmgang::Grader that grades the end state; its simulator registry also runs the world's calls.

run_case_f

my $result = await $runner->run_case_f( $case, repetition => 2, seed => 2 );

Plays one Langertha::Holmgang::Case (see "DESCRIPTION"). seed (integer, optional) goes unchanged into every request of the case; repetition (default 1) is only carried into the result. no_vision (optional) is the reason the vision pre-check of the suite gave ("vision_check_f" in Langertha::Holmgang::Target): a case with images then ends no_vision with that text as its error, without a request and without a turn; a case without images ignores it. Returns a Future of the result, the per-repetition shape "add_run" in Langertha::Holmgang::Report takes (ADR 0006), plain data:

  • case_id, repetition, seed

  • status, score, error, rules, diff, sim_gaps, forbidden_calls, injection_held -- from "grade" in Langertha::Holmgang::Grader. error is the target's error text when it did not answer, scrubbed ("scrub" in Langertha::Holmgang::Target), undef otherwise (a turn limit has none). A failed request ends the case: its text never goes back to the model.

  • turns -- one entry per model call that reached the target, the failed one included: input_tokens, output_tokens, reasoning_tokens (from Langertha::Usage, undef when the target sent none), duration_ms (wall time of the call), finish_reason, calls (tool calls of the turn).

  • calls -- every tool call in order: tool, args (decoded), succeeded, sim_gap.

  • transcript -- the conversation without the system prompt, in the engine's wire format, ending with the final assistant text.

  • answer -- the last text of the model (raw, only to look at), and state -- the end state. Neither goes into a report.

run_case

my $result = $runner->run_case( $case, %options );

"run_case_f", waited for.

mock_engine

my $engine = Langertha::Holmgang::Runner->mock_engine( $case, 'solution' );   # or 'text'

A Langertha::Engine::Virtual as the mock model of the lint (and of dry runs): solution calls the reference solution, one call per turn, then answers Done.; text answers with text only, at once. For a case with images it claims image_input (Langertha::Role::ImageInput), answers OK once its script is done (which a case never reaches) and reports no token usage, so "vision_check_f" in Langertha::Holmgang::Target lets it through unmeasured. It replaces cai-platform's Eval::FakeTarget (ADR 0005); unlike that one it estimates no tokens and emulates no context window (the target's error path is tested with "error" in Langertha::Virtual::Rules reactions instead).

lint_mock_runs

my $lint = Langertha::Holmgang::Lint->new( mock_runs => Langertha::Holmgang::Runner->lint_mock_runs );

The code reference for "mock_runs" in Langertha::Holmgang::Lint: the two mock runs of the lint (cai-platform docs/entwurf-evals.md §3.5), each a real "run_case" over a "mock_engine", graded with the lint's grader.

  • Replaying the reference solution passes, twice identically -- status, score, rules, diff, calls, gaps, transcript, end state and the turns without their wall time.

  • Answering text only fails -- except in the "no tool" family (solution: []), where it must pass.

A run that does not hold throws a Langertha::Holmgang::Error on the field solution or rules; the case does not load.

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.