NAME
holmgang - Lint Holmgang suites, run them against a Langertha engine, evaluate the reports
VERSION
version 0.001
SYNOPSIS
holmgang lint /srv/suites/demo-fan /srv/more-suites
LANGERTHA_OPENAI_API_KEY=... holmgang run --suite /srv/suites/demo-fan \
--target OpenAI --model gpt-4o-mini --repeat 3 --report /data/reports/run.json
holmgang run --suite /srv/suites/demo-fan --target vLLM \
--url http://10.0.0.5:8000/v1 --model some-model \
--variant max_tokens=4000 --variant enable_thinking=off --concurrency 4
# two variants of one target, side by side in the report
holmgang run --suite /srv/suites/demo-fan --target vLLM --url http://10.0.0.5:8000/v1 \
--variant think-off:enable_thinking=off \
--variant think-on:enable_thinking=on --variant think-on:deadline=1800 \
--report /data/reports/run.json
# continue a run that was cut off: only the missing results are played
holmgang run ... --resume /data/reports/run.json --report /data/reports/run.json
holmgang run --suite /srv/suites/demo-fan --target mock-solution # dry run, no network
holmgang report /data/reports/*.json --by topic --format markdown --suite /srv/suites/demo-fan
# a suite in the case DSL as format 2, and with a private seed for its generators
holmgang export-yaml /srv/dsl-suites/example-sums --out /srv/exported
holmgang run --suite /srv/dsl-suites/example-sums --target mock-solution --seed example-sums=90210
DESCRIPTION
The command line of Langertha::Holmgang (ADR 0009). Four commands:
lint loads each suite exactly as a run would -- format 2 shape checks, lint and its two mock runs -- and prints ok with every case, or FAIL with the refusal, which names the case file and the field. An argument is a suite directory (it holds suite.yaml, or suite.pl in the case DSL) or a directory of suites.
run plays every case of the suites against one target, under each of its variants, repetition after repetition, with at most --concurrency requests open at once (Langertha::Holmgang::Batch); it prints one line per case and repetition as each ends and then the evaluation per topic. With --report it writes the JSON report (ADR 0006), also while the run goes on, so a run that is cut off can be continued with --resume. Write reports outside any checkout: a report quotes its cases. SIGINT or SIGTERM cancels: no further case starts, the cases in play finish, the report is written (the run cancelled); a second signal ends the process at once.
report reads report files, joins them, and prints the evaluation per topic: the comparison of targets, the verdicts per suite and, where cases have a length tier or a difficulty, the curves per tier and per difficulty.
export-yaml writes each suite in the case DSL (suite.pl) among its arguments -- suite directories or directories of suites -- as suite format 2 into --out/id: suite.yaml, cases/*.yaml and the images. The suite is loaded first, lint included, and the export must load back with the same digest; a format 2 suite is skipped. A suite directory that already exists under --out is a usage error before anything is written: nothing is overwritten.
Exit codes: 0 everything loaded or ran and every verdict is ok; 1 a suite was refused (lint, export-yaml), a verdict is not ok (run, report; a suite in which every result is error measured nothing, its verdict reads NOT MEASURED, with or without a threshold), or the target answered no case of the run -- every result error, context_exceeded or no_vision (run; said on standard error with the count per status and the first error text, the report is written all the same); 2 a usage or setup error, or a run that aborted or was cancelled.
Keys: holmgang reads no key. An engine takes its key from where Langertha reads it, its environment variable (LANGERTHA_OPENAI_API_KEY, LANGERTHA_ANTHROPIC_API_KEY ...). There is no default target and no default endpoint.
--suite DIR
run, report: a suite directory, repeatable. For report only the suites' threshold is taken; without --suite a verdict is judged without one: by its hard cases, an unfinished run and whether anything was measured.
--target NAME
run, required: a Langertha engine -- OpenAI is Langertha::Engine::OpenAI, a name with :: a class name as written -- or one of the two network-free targets: mock-solution replays every case's reference solution, mock-text answers text only.
--model M
run: the engine's model.
--url U
run: the engine's url.
--option K=V
run, repeatable: any other constructor argument of the engine. A value that parses as JSON (a number, true, an object ...) is taken as that, otherwise as text. api_key is refused -- the command line is visible to every user of the host.
--label L
run: the target's name in the report and the tables; default the model, else the engine name.
--variant K=V
--variant NAME:K=V
run, repeatable: the variants of the target, each a name and its settings, values as for --option. K=V adds to the one unnamed variant, whose name is its sorted settings (max_tokens=4000,temperature=0); NAME:K=V adds to the variant NAME, NAME: names one without settings. Every variant runs every suite; the report has one run per variant and suite. Without --variant there is one variant, default. The settings are named and checked ("check_variant" in Langertha::Holmgang::Target):
enable_thinking=on|off-- sent aschat_template_kwargs.enable_thinking; only to an OpenAI-compatible engine.reasoning_effort=none|minimal|low|medium|high|xhigh|max-- only to an engine that takes it.max_tokens=N-- 1 to 9999999, beats the case's.temperature=T-- 0 to 2, beats the case's; only to an engine that takes it.deadline=S-- seconds one request may take (above 0, at most 86400); a request that runs out ends the caseerrorwithtimeout: deadline (S s). Needs the engine on an event loop (IO::Async and Net::Async::HTTP installed). A timeout without data is the engine'suser_agent_timeout:--option user_agent_timeout=900.
Anything else is refused, and so is a setting the engine would drop.
--concurrency N
run: at most N requests open against the target at once, 1 to 64; default 1. Each of the N slots gets its own engine (and connection).
--resume FILE
run: continue the run of an earlier report: every result it holds is taken over, only the missing ones are played. The report must come from the same suites (unchanged), target, variants and --repeat; anything else is refused. --report may name the same file.
--repeat N
run: repetitions per suite, 1 to 100; default the suite's repeat, else 1.
--report FILE
run: write the JSON report there.
--seed ID=N
lint, run, export-yaml, repeatable: replace the seed of every generate block of the DSL suite ID with N (0 to 4294967295), so a private measurement seed never has to be written into the suite file. An id that is no suite in the case DSL among the command's suites is a usage error. Another seed is another suite, with another digest.
--out DIR
export-yaml, required: the directory the suites are written into, one subdirectory per suite id; created when missing.
--by topic
report: the axis of the evaluation; topic is the only one.
--format F
report: text (default), markdown or json.
SEE ALSO
SUPPORT
Issues
Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.
IRC
Join #langertha on irc.perl.org or message Getty directly.
CONTRIBUTING
Contributions are welcome! Please fork the repository and submit a pull request.
AUTHOR
Torsten Raudssus <torsten@raudssus.de>
COPYRIGHT AND LICENSE
This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.