NAME

Langertha::Holmgang - Deterministic tool-use benchmarks for language models via Langertha

VERSION

version 0.001

SYNOPSIS

# lint a suite in format 2: shape checks, the self-check of every case, two mock runs
holmgang lint /data/suites/demo-motd

# a dry run without network: the mock model replays every reference solution
holmgang run --suite /data/suites/demo-motd --target mock-solution --report /data/reports/dry.json

# a real target: any Langertha engine; the key comes from the engine's environment variable
holmgang run --suite /data/suites/demo-motd --target vLLM \
  --url http://10.0.0.5:8000/v1 --model some-model --repeat 3 --report /data/reports/run.json

# evaluate reports per topic
holmgang report /data/reports/*.json --suite /data/suites/demo-motd --format markdown

DESCRIPTION

Langertha::Holmgang measures language models on tasks with tools and grades them deterministically: no judge model and no text comparison. A case is a task in a simulated world; the model works on it through general tools with real syntax, and the case passes only when the world ends up in the right state. Any engine that Langertha speaks can be measured.

A holmgang was a Norse duel under fixed rules: a marked-out ground, witnesses, agreed weapons, and a verdict everyone accepted.

STATUS

Usable today: suites in format 2 (suite.yaml plus cases/*.yaml) and in the case DSL of this module (suite.pl, "THE CASE DSL") with the loader and its lint, the simulators, the grader, the runner over any Langertha engine, several variants per target, parallel requests and resume (Langertha::Holmgang::Batch), the JSON report, the evaluation per topic, the YAML export of a DSL suite ("export" in Langertha::Holmgang::Loader), and the holmgang command line (lint, run, report, export-yaml).

The distribution ships no real benchmark cases: published cases end up in the training data of the next models.

THE CASE DSL

use Langertha::Holmgang;

suite 'example-sums' => sub {
  title   'Example: add numbers, hand in the sum';
  version 1;
  topic   'reasoning';
  system_prompt 'Use the tool to hand in your answer. Then reply briefly without a tool.';

  tool submit_answer => (
    description => 'Hand in the final answer. A later call replaces an earlier one.',
    parameters  => { type => 'object', required => ['value'],
                     properties => { value => { type => 'integer' } } },
    handlers    => [ handler apply => [ { set => 'answer', value => '{value|int}' } ], output => 'ok' ],
  );

  generate seed => 4711, sub {
    my ( $rng ) = @_;
    for my $n ( 1 .. 3 ) {
      my @terms = map { $rng->int( 10, 99 ) } 1 .. 4;
      my $sum = 0;
      $sum += $_ for @terms;
      case sprintf( 'sum-%02d', $n ) => sub {
        title      'Sum of four numbers, variant '.$n;
        difficulty 'leicht';
        world answer => undef;
        ask 'Add '.join( ', ', @terms ).' and hand in the sum.';
        rule 'sum handed in'      => at 'answer', equals => $sum;
        rule 'nothing else moved' => unchanged_except ['answer'];
        solution    call submit_answer => { value => $sum };
        alternative 'corrected answer' =>
          call( submit_answer => { value => $sum - 1 } ), call( submit_answer => { value => $sum } );
        wrong 'off by one' => call submit_answer => { value => $sum + 1 };
      };
    }
  };
};

use Langertha::Holmgang exports the keywords below (and, as every Moose::Exporter module does, turns on strict and warnings); no Langertha::Holmgang removes them. A keyword can be renamed on import, for a file that also uses a module exporting the same name:

use Langertha::Holmgang call => { -as => 'tool_call' };

The DSL is sugar over suite format 2 (ADR 0004, ADR 0007): every keyword writes a format 2 key, and when the suite block closes the suite goes through the same structure checks, the same lint with both mock runs and the same runner as a suite.yaml ("assemble" in Langertha::Holmgang::Loader). A refusal names the file, the case and the field like the loader's: suite.pl: case sum-01: rules[0].equals. A keyword outside its block, a single-valued keyword given twice, or an unknown key in a keyword's pairs dies at the line that wrote it.

Where DSL suites live

A suite directory in a search directory holds either suite.yaml with cases/ or a suite.pl; both is refused. suite.pl declares exactly one suite, whose id is the directory name; texts for "file" and images sit beside it. The loader runs the file in a package of its own: it is code, with the trust of an installed module, so load it only from directories you control. Outside the loader -- in a test or a script -- suite returns the Langertha::Holmgang::Suite as well, and paths are relative to the calling file.

Cases are ordered by the name of their export file, <id>.yaml -- the order format 2 loads case files in, so a-b comes before a; an id given twice dies.

The YAML export

holmgang export-yaml /srv/dsl-suites/example-sums --out /srv/exported

writes the suite as suite format 2 -- suite.yaml, cases/<id>.yaml and the images -- so its cases can be read and compared without Perl ("export" in Langertha::Holmgang::Loader). Texts read by file are inline, haystack specs stay generate specs, generated cases are plain cases. The export loads into the same suite, and a DSL suite's digest is the digest of its export: a run of the one is a run of the other.

Two kinds of keywords

Declarations (suite, case, title, rule, device, ...) record into the innermost open suite or case block and return nothing. Constructors (at, call, run, file, handler, ...) record nothing and return a piece of format 2 data, so they can be used anywhere, also in variables and loops. Fixed-arity constructors carry a prototype -- call($$), run($$), put($$$), file($) -- so several in one list are written with parentheses: run( a => 'x' ), run( b => 'y' ).

The suite

suite ID => sub { ... }

The suite; id. Returns the loaded Langertha::Holmgang::Suite.

title, description, version, topic, license, repeat

The format 2 key of the same name, once each. title and source are also case keywords: they belong to the innermost open block.

threshold pass_rate => ..., pass_k => ...
source title => ..., authors => ..., license => ..., url => ..., location => ...
system_prompt TEXT

defaults.system in a suite, system in a case.

settings temperature => ..., max_tokens => ..., max_turns => ...

defaults.* in a suite, the case keys in a case; each key once.

tool NAME => ( description => ..., parameters => {...}, simulator => ... | handlers => [...] )

One entry of tools. parameters is the JSON Schema subset of Langertha::Holmgang::ToolSchema, as data (type lists and const included).

handler match => ..., when => [...], apply => [...], output => ..., error => 1

Constructor of one handler. The apply operations (set, push, delete ...) stay data hashes: push and delete are Perl builtins.

case ID => sub { ... }

One case, in a suite or a generate block.

generate seed => N, sub { my ( $rng ) = @_; ... }

Runs its block once, while the suite is built, with a Langertha::Holmgang::Random seeded with N: the same seed gives the same cases on every perl and platform. The block declares ordinary cases; nothing about the seed reaches the structure. Never use rand or srand in it, and never let the order of keys %hash pick a case. The loader can replace the seed per suite id ("seeds" in Langertha::Holmgang::Loader; on the command line --seed ID=N), so a private measurement seed never has to be written into a public file.

The case

title, difficulty, weight

The case key of the same name, once each (difficulty: leicht, mittel, schwer; weight: a positive number).

tags LIST

Appends to tags.

tier '48k'

The length tier: the tag stufe-48k; one per case.

ask CONTENT, untrusted => 1, images => [ ... ]
message ROLE => CONTENT, untrusted => 1, images => [ ... ]

A message (ask: role: user). CONTENT is a text, a { generate => ... } spec ("haystack"), an "excerpt" or a list of parts. images (only on a user message) lists file names relative to the suite directory, or "image" entries; a plain file name 'img/a.png' is { file => 'img/a.png' }.

image FILE, untrusted => 1

Constructor of one image entry, { file, untrusted }.

device NAME => TYPE => { ... }

state.NAME with type: TYPE; the type is checked against the simulator registry when the suite closes, so a plugin only has to be registered by then.

world NAME => VALUE

state.NAME without a type (handler worlds).

observe match => { ... }, when => [ ... ], output => ..., error => 1

One entry of observe.

rule TITLE => KIND

One rule. KIND is one of the rule constructors, each KIND VALUE, %rest with the format 2 pairs of that kind and required => 0: at PATH, %expectations (the rule kind state; state is a Perl feature keyword), unchanged_except [PATHS], calls [CALLS], not_called CALL, only_tools [NAMES], check { host, command }, expect => { ... }, packet { ... }, verdict => ... and login { ... }, result => .... Expectations are pair keys (equals, tolerance, in, regex, any, before ...); a regex may be a qr//, stored as its string form.

solution CALLS

The reference solution; solution() is [], the "no tool" family.

alternative TITLE => CALLS, wrong TITLE => CALLS

One entry of alternatives, of wrong_solutions.

Calls and contents

call TOOL => { ARGS }

A call, { tool, args }. In calls and not_called the arguments are patterns, as in format 2. A parameter of type: boolean takes a boolean (JSON::PP::true), not 1.

run HOST => COMMAND, put HOST => PATH => CONTENT

A call of the suite's one tool with simulator: run_command (write_file); resolved when the suite closes, which dies unless exactly one such tool exists.

excerpt 'quellen/NAME.txt', from => LABEL, to => LABEL

A message part with the provisions from to to (such as '§ 305', 'Artikel 28') of a statute text under quellen/ of the suite directory: the format 2 part { auszug => { datei, von, bis } } (Langertha::Holmgang::Excerpt), set in when the suite closes. Use it alone or in a list of parts: ask [ "The statute:\n\n", excerpt( ... ), "\n\nThe contract: ..." ]. The YAML export writes the part as it is and copies the fixture.

file 'NAME'

The text of a file, relative to the suite file's directory.

haystack kind => ..., tokens => ..., seed => ..., insert => [ ... ], exclude => [ ... ]

Generated long content, { generate => { ... } } (Langertha::Holmgang::Haystack); its keys are checked when the suite closes.

Names that collide

state (a feature keyword) is at for a rule and device/world for the world; system is system_prompt; write is put. version shadows the core package in the suite file: write version::->parse there if needed. Test2::V0 exports call and check, Path::Class file: rename on import (above).

EXAMPLE SUITES

Six small, invented suites to learn from ship in the distribution's share directory (ADR 0010): demo-linux (a Linux device), demo-lamp (a world without devices, its tool described by handlers), demo-haystack (generated input), demo-sehen (an image on a user message, handed in through the answer world sehen), demo-roost (a typed answer world plugin, an invented dragon roost) -- all five in format 2 -- and demo-beacons, a suite.pl in the case DSL whose three cases a generate block builds from a seed (an invented chain of hill beacons). In a checkout they are under share/suites; after installation under

perl -MFile::ShareDir=dist_dir -E 'say dist_dir("Langertha-Holmgang")."/suites"'

holmgang lint "$(perl -MFile::ShareDir=dist_dir -E 'say dist_dir("Langertha-Holmgang")')/suites"

Details in "Example suites" in Langertha::Holmgang::Loader.

HOW IT FITS TOGETHER

1. Load. Langertha::Holmgang::Loader reads a suite directory in format 2 (the format: "SUITE FORMAT 2" in Langertha::Holmgang::Loader), checks every file's shape (Langertha::Holmgang::Structure) and lints every case (Langertha::Holmgang::Lint). The result is a Langertha::Holmgang::Suite with its Langertha::Holmgang::Cases. A case that fails lint does not load.
2. Simulate. Each case starts a fresh Langertha::Holmgang::World from its initial state. A tool call goes to the case's observations, then to the simulator of the device's type (Langertha::Holmgang::Simulators, plugins composing Langertha::Holmgang::Role::Simulator or Langertha::Holmgang::Role::AnswerWorld), or to the tool's handlers. Simulators touch only their state: no shell-out, no socket, no host file.
3. Run. Langertha::Holmgang::Runner plays the case against a Langertha::Holmgang::Target — a Langertha engine plus variant settings — turn by turn until the model answers without a tool call, max_turns is reached, or the target fails. The model sees only the system prompt, the messages and the tool schemas; never the state, the observations, the rules, the solutions or the untrusted marks.
4. Grade. Langertha::Holmgang::Grader checks the rules at the end state and on the calls ("RULES" in Langertha::Holmgang::Grader) and gives each repetition one status ("STATUSES" in Langertha::Holmgang::Grader): passed, failed, uncertain, turn_limit, error, context_exceeded or no_vision.
5. Report. Langertha::Holmgang::Report writes what was measured as one JSON document; Langertha::Holmgang::Evaluation derives rates per topic, weights, curves per length tier, verdicts per suite and the comparison of targets from one or more reports.

The command line ties these together: holmgang (Langertha::Holmgang::CLI).

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.