NAME

holmgang - Lint Holmgang suites, run them against a Langertha engine, evaluate the reports

VERSION

version 0.001

SYNOPSIS

holmgang lint /srv/suites/demo-fan /srv/more-suites

LANGERTHA_OPENAI_API_KEY=... holmgang run --suite /srv/suites/demo-fan \
  --target OpenAI --model gpt-4o-mini --repeat 3 --report /data/reports/run.json

holmgang run --suite /srv/suites/demo-fan --target vLLM \
  --url http://10.0.0.5:8000/v1 --model some-model \
  --variant max_tokens=4000 --variant enable_thinking=off --concurrency 4

# two variants of one target, side by side in the report
holmgang run --suite /srv/suites/demo-fan --target vLLM --url http://10.0.0.5:8000/v1 \
  --variant think-off:enable_thinking=off \
  --variant think-on:enable_thinking=on --variant think-on:deadline=1800 \
  --report /data/reports/run.json

# continue a run that was cut off: only the missing results are played
holmgang run ... --resume /data/reports/run.json --report /data/reports/run.json

holmgang run --suite /srv/suites/demo-fan --target mock-solution   # dry run, no network

holmgang report /data/reports/*.json --by topic --format markdown --suite /srv/suites/demo-fan

# a suite in the case DSL as format 2, and with a private seed for its generators
holmgang export-yaml /srv/dsl-suites/example-sums --out /srv/exported
holmgang run --suite /srv/dsl-suites/example-sums --target mock-solution --seed example-sums=90210

DESCRIPTION

The command line of Langertha::Holmgang (ADR 0009). Four commands:

lint loads each suite exactly as a run would -- format 2 shape checks, lint and its two mock runs -- and prints ok with every case, or FAIL with the refusal, which names the case file and the field. An argument is a suite directory (it holds suite.yaml, or suite.pl in the case DSL) or a directory of suites.

run plays every case of the suites against one target, under each of its variants, repetition after repetition, with at most --concurrency requests open at once (Langertha::Holmgang::Batch); it prints one line per case and repetition as each ends and then the evaluation per topic. With --report it writes the JSON report (ADR 0006), also while the run goes on, so a run that is cut off can be continued with --resume. Write reports outside any checkout: a report quotes its cases. SIGINT or SIGTERM cancels: no further case starts, the cases in play finish, the report is written (the run cancelled); a second signal ends the process at once.

report reads report files, joins them, and prints the evaluation per topic: the comparison of targets, the verdicts per suite and, where cases have a length tier or a difficulty, the curves per tier and per difficulty.

export-yaml writes each suite in the case DSL (suite.pl) among its arguments -- suite directories or directories of suites -- as suite format 2 into --out/id: suite.yaml, cases/*.yaml and the images. The suite is loaded first, lint included, and the export must load back with the same digest; a format 2 suite is skipped. A suite directory that already exists under --out is a usage error before anything is written: nothing is overwritten.

Exit codes: 0 everything loaded or ran and every verdict is ok; 1 a suite was refused (lint, export-yaml), a verdict is not ok (run, report; a suite in which every result is error measured nothing, its verdict reads NOT MEASURED, with or without a threshold), or the target answered no case of the run -- every result error, context_exceeded or no_vision (run; said on standard error with the count per status and the first error text, the report is written all the same); 2 a usage or setup error, or a run that aborted or was cancelled.

Keys: holmgang reads no key. An engine takes its key from where Langertha reads it, its environment variable (LANGERTHA_OPENAI_API_KEY, LANGERTHA_ANTHROPIC_API_KEY ...). There is no default target and no default endpoint.

--suite DIR

run, report: a suite directory, repeatable. For report only the suites' threshold is taken; without --suite a verdict is judged without one: by its hard cases, an unfinished run and whether anything was measured.

--target NAME

run, required: a Langertha engine -- OpenAI is Langertha::Engine::OpenAI, a name with :: a class name as written -- or one of the two network-free targets: mock-solution replays every case's reference solution, mock-text answers text only.

--model M

run: the engine's model.

--url U

run: the engine's url.

--option K=V

run, repeatable: any other constructor argument of the engine. A value that parses as JSON (a number, true, an object ...) is taken as that, otherwise as text. api_key is refused -- the command line is visible to every user of the host.

--label L

run: the target's name in the report and the tables; default the model, else the engine name.

--variant K=V

--variant NAME:K=V

run, repeatable: the variants of the target, each a name and its settings, values as for --option. K=V adds to the one unnamed variant, whose name is its sorted settings (max_tokens=4000,temperature=0); NAME:K=V adds to the variant NAME, NAME: names one without settings. Every variant runs every suite; the report has one run per variant and suite. Without --variant there is one variant, default. The settings are named and checked ("check_variant" in Langertha::Holmgang::Target):

  • enable_thinking=on|off -- sent as chat_template_kwargs.enable_thinking; only to an OpenAI-compatible engine.

  • reasoning_effort=none|minimal|low|medium|high|xhigh|max -- only to an engine that takes it.

  • max_tokens=N -- 1 to 9999999, beats the case's.

  • temperature=T -- 0 to 2, beats the case's; only to an engine that takes it.

  • deadline=S -- seconds one request may take (above 0, at most 86400); a request that runs out ends the case error with timeout: deadline (S s). Needs the engine on an event loop (IO::Async and Net::Async::HTTP installed). A timeout without data is the engine's user_agent_timeout: --option user_agent_timeout=900.

Anything else is refused, and so is a setting the engine would drop.

--concurrency N

run: at most N requests open against the target at once, 1 to 64; default 1. Each of the N slots gets its own engine (and connection).

--resume FILE

run: continue the run of an earlier report: every result it holds is taken over, only the missing ones are played. The report must come from the same suites (unchanged), target, variants and --repeat; anything else is refused. --report may name the same file.

--repeat N

run: repetitions per suite, 1 to 100; default the suite's repeat, else 1.

--report FILE

run: write the JSON report there.

--seed ID=N

lint, run, export-yaml, repeatable: replace the seed of every generate block of the DSL suite ID with N (0 to 4294967295), so a private measurement seed never has to be written into the suite file. An id that is no suite in the case DSL among the command's suites is a usage error. Another seed is another suite, with another digest.

--out DIR

export-yaml, required: the directory the suites are written into, one subdirectory per suite id; created when missing.

--by topic

report: the axis of the evaluation; topic is the only one.

--format F

report: text (default), markdown or json.

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.