NAME

Langertha::Holmgang::Target - A Langertha engine as a benchmark target: variant settings, one model turn, failed turns classified

VERSION

version 0.001

SYNOPSIS

use Langertha::Holmgang::Target;
use Langertha::Engine::Virtual;
use Langertha::Virtual::Rules qw( answer error );

my $target = Langertha::Holmgang::Target->new(
  engine  => Langertha::Engine::Virtual->new( rules => [ qr/./ => answer('done') ] ),
  variant => { max_tokens => 4000 },     # chat_f options, passed through unchecked
);

my $turn = $target->chat_turn_f(
  messages    => [ { role => 'user', content => 'Fix the route on gw-a.' } ],
  tools       => \@tool_schemas,
  temperature => 0,
)->get;

if ( defined $turn->{error_class} ) {
  # 'no_vision', 'context_exceeded' or 'error' -- the case ends with that status
  warn $turn->{error};
}
else {
  my $response = $turn->{response};      # Langertha::Response
}

DESCRIPTION

A target is what a run measures (CONTEXT.md): one Langertha engine -- any engine, Langertha::Engine::Virtual in tests -- plus the settings of the variant under which it runs. The runner asks it for one model turn at a time and gets a normalised result back: either the engine's Langertha::Response, or the error text and its class.

The class decides the case status of ADR 0006 when the target did not answer: no_vision (the target cannot see an image of the case), context_exceeded (the request does not fit its context window) or error (any other failure). Langertha has no structured request failure yet, only the text of the croak (<class> request failed: <status line> - <body>), so the class is read off that text with the signatures of the common servers; see "classify_error".

engine

Required. The Langertha engine the turns go to -- any object with chat_f (Langertha::Role::Chat). Its own attributes (model, url, system prompt, engine-level controls) stay as they are; the target never changes it.

variant

Optional, default {}. The variant's settings as chat_f options (max_tokens, temperature, reasoning_effort, chat_template_kwargs ...). They are added to every turn and win over the turn's own options of the same name, as cai-platform's variant max_tokens beats the case's. A plain HashRef, passed through unchecked: this is the wire-level form. The named and checked settings of a variant go through "for_variant", which builds this HashRef from them.

deadline

Optional. Seconds one turn may take, a number above 0. When the engine runs on an event loop (async_loop), "chat_turn_f" races the request against a timer on that loop; a turn that runs out fails with timeout: deadline (N s) (class error) and the request is cancelled. On an engine without a loop the request has already finished when chat_f returns, so there is nothing to race; "check_variant" refuses a deadline for such an engine.

variant_keys

my @keys = Langertha::Holmgang::Target->variant_keys;

The settings a variant may name: enable_thinking, reasoning_effort, max_tokens, temperature, deadline.

check_variant

my $settings = Langertha::Holmgang::Target->check_variant( \%settings );
my $settings = Langertha::Holmgang::Target->check_variant( \%settings, $engine );

Checks the settings of a variant and returns them normalised. A variant names only these, each checked (cai-platform ModelTester::check_variant):

  • enable_thinking -- on/off, true/false, 1/0 or a JSON boolean; normalised to a JSON boolean. Goes out as chat_template_kwargs => { enable_thinking => ... } (Qwen, Nemotron, Gemma under llama.cpp, vLLM, SGLang).

  • reasoning_effort -- one of Langertha's values none minimal low medium high xhigh max.

  • max_tokens -- a whole number from 1 to 9999999; beats the case's.

  • temperature -- a number from 0 to 2; beats the case's.

  • deadline -- seconds per turn, above 0 and at most 86400 ("deadline").

An empty value names nothing and is left out. With an $engine the settings are also checked against it: temperature and reasoning_effort only where it supports them (Langertha would drop them with a warning, and the report would name a setting that never went out), enable_thinking only on an engine with Langertha::Role::OpenAICompatible, deadline only on an engine with an event loop. Throws a Langertha::Holmgang::Error on the field variant.<key>.

for_variant

my $target = Langertha::Holmgang::Target->for_variant(
  engine   => $engine,
  settings => { enable_thinking => 'off', max_tokens => 4000, deadline => 1800 },
);

A target for one variant: the settings checked against the engine ("check_variant") and turned into "variant" (the chat_f options) and "deadline".

error_body_max_length

Langertha::Holmgang::Target->error_body_max_length;   # 65536

How many characters of a failed response's body the error text keeps, at least. Langertha cuts the body in its croak at $Langertha::Role::HTTP::error_body_max_length (500); "chat_turn_f" raises that to this value with local for the duration of the call (a higher value already set stays). Limit: the local covers the synchronous transports -- Langertha::Engine::Virtual and the LWP fallback without Net::Async::HTTP, where the request and its croak happen inside the call. With a real event loop the croak happens later, outside the local, and the body is cut at Langertha's limit again; signatures in the first 500 characters are still found. An uncut body is a Langertha gap.

classify_error

$target->classify_error($error_text);   # 'no_vision' | 'context_exceeded' | 'error'

The class of a failed turn from its error text (also callable on the class). Only ever asked for a turn that failed: a response that merely mentions one of the phrases is a response, never one of these classes.

The first that matches wins, in the order of ADR 0006:

  • no_vision -- llama.cpp without a projector ("image input is not supported ... mmproj"), vLLM ("is not a multimodal model").

  • context_exceeded -- OpenAI (context_length_exceeded), vLLM ("maximum context length", "longer than the maximum model length", max_model_len), llama.cpp (exceed_context_size_error, "exceeds the available context size"), SGLang ("longer than the model's context length"), TGI ("inputs tokens + max_new_tokens must be <= N"), LiteLLM (ContextWindowExceededError), Ollama ("input length exceeds the context length"), Anthropic ("prompt is too long"), and "context window exceeded" / "is too small". The expression is cai-platform's $CONTEXT_RE unchanged.

  • error -- everything else, an empty text included: a 500 without a hint, a timeout, a refused connection, a croak before the request.

scrub

my $clean = $target->scrub($text);    # also callable on the class, without the key

The text of a failed turn with what must not leave the run removed (ADR 0006: "error texts pass the engine's scrubbing first" -- Langertha has no scrubber of its own, so the target does it). "chat_turn_f" scrubs its error; the class of the failure is read off the raw text before. What goes:

  • the engine's own api_key (4 characters and up), wherever it stands -- ***; only on an object, the class knows no key;

  • die and Carp locations (at FILE line N., called at ..., with , <FH> line N when Perl adds it), dropped;

  • URL credentials: the userinfo (https://***@host/...) and the value of a query parameter named key, api_key, access_token, token, auth, secret, password, sig, signature -- ***, the rest of the URL stays;

  • header and JSON field values: Authorization (after its scheme), Bearer tokens, x-api-key, x-goog-api-key, api_key/api-key -- ***;

  • key shapes of the common providers (sk-, pplx-, gsk_, xai-, hf_ with 16 characters and up, AIza...) -- ***;

  • absolute host paths (/a/b, ~/a, C:\a\b; not the path of a URL, not a single segment) -- <path>.

The status line, the provider's reason and the URL's host and path stay readable. This is a net for what Langertha's error text can carry, not a proof: a credential the engine does not hold and that has none of these shapes passes.

sight_test_min_tokens

Langertha::Holmgang::Target->sight_test_min_tokens;   # 200

How many input tokens the sight test image ("sight_test_image") must add to a request before "vision_check_f" counts it as seen. Common vision models spend 256 (Gemma 3, InternVL) to over 1000 (Pixtral) on a 512 by 512 image; a template that drops the image adds next to nothing.

sight_test_image

my $png = Langertha::Holmgang::Target->sight_test_image;   # bytes

The image of the sight test: a 512 by 512 grey PNG with a checkerboard, built in memory, the same bytes on every call.

vision_check_f

my $check = await $target->vision_check_f;
# { no_vision => undef | $reason, check => 'wire' | 'probe' | 'sight_test',
#   added_tokens => n, note => $text }

The pre-check of a target for a suite whose cases carry images (k27, ADR 0006): can the target see an image? A pre-check, never a graded case. In this order, the first that decides wins:

1. The wire. An engine that does not compose Langertha::Role::ImageInput has no image part Langertha vouches for: no_vision. (Langertha::Engine::Virtual is one; the runner's "mock_engine" in Langertha::Holmgang::Runner claims it for a case with images.)
2. The server's model metadata. "probe_model_capabilities_f" in Langertha::Role::Capabilities asks the providers Langertha can ask (llama.cpp /props, Ollama, LM Studio, OpenRouter, Mistral, TSystems; on every other engine it asks nothing). When it learned image_input => 0 for the engine's chat_model: no_vision. A failing probe decides nothing. A static "no claim" (vLLM, llama.cpp before the probe, gateways) decides nothing either -- the model behind them is unknown to the client.
3. The sight test. Two short requests through "chat_turn_f" (the variant applies): a text, then the same text with "sight_test_image". If the second fails with a no_vision signature ("classify_error"): no_vision. If the image added fewer input tokens than "sight_test_min_tokens", the chat template dropped it without a word: no_vision. A failed request or a target without token usage decides nothing (note says why); the cases then run and show what they get.

The Future is always done. no_vision is undef when the target may see the image, else the reason, which the runner gives every case with images of the suite as its error ("run_case_f" in Langertha::Holmgang::Runner, option no_vision).

chat_turn_f

my $turn = await $target->chat_turn_f(
  messages => \@messages,      # system prompt included as the runner builds it
  tools    => \@tools,         # the suite's tool schemas
  %controls,                   # temperature, max_tokens, seed, ...
);

One model turn: the options go to the engine's chat_f unchanged, with the "variant" settings added on top. Returns a Future that is always done -- a failed request is a result, not a failed Future -- with a HashRef:

  • response -- the Langertha::Response of a turn that succeeded, undef otherwise. Usage, finish reason, tool calls and timing are read from it.

  • error -- the error text of a failed turn (Langertha's croak, without the trailing whitespace, through "scrub"), undef otherwise.

  • error_class -- undef for a turn that succeeded, else no_vision, context_exceeded or error ("classify_error"); the runner ends the case with that status.

With a "deadline", a request still running after that many seconds is cancelled and the turn fails with timeout: deadline (N s) (class error).

Anything chat_f throws or fails with counts as a failed turn, including a croak before the request (a capability exclusion, a malformed tool). What the model may see is the caller's business: the target strips nothing, so the runner hands it messages and tools without state, rules or untrusted marks. The error body is kept up to "error_body_max_length" characters (see there for the limit under a real event loop).

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.