NAME

Langertha::Holmgang::CLI - The holmgang command: lint suites, run them against a Langertha engine, evaluate reports

VERSION

version 0.001

SYNOPSIS

use Langertha::Holmgang::CLI;

exit Langertha::Holmgang::CLI->new->run(@ARGV);

# in a test: capture the output, get the exit code
open my $out, '>', \my $text;
my $code = Langertha::Holmgang::CLI->new( out => $out, err => $out )
  ->run( lint => '/tmp/suites/demo' );

DESCRIPTION

The commands behind bin/holmgang (ADR 0009): lint loads suites as the loader does, run plays them against a target and writes a report, report evaluates report files per topic, export-yaml writes DSL suites as suite format 2. Every command returns its exit code instead of exiting: 0 all well, 1 a suite refused, a verdict not ok, or a run in which the target answered no case, 2 a usage or setup error. The options are described in holmgang.

The CLI reads no key: a real engine takes its key from where Langertha reads it (its environment variable). There is no default target.

out

The handle for the command's output; standard output by default.

err

The handle for usage and setup errors; standard error by default.

loader

The Langertha::Holmgang::Loader every suite goes through.

checkpoint_seconds

How often run --report writes the report while the batch runs, at most: after a result, when this many seconds have passed since the last write. Default 10; 0 writes after every result. The checkpoint is what --resume continues after a crash.

run

my $code = $cli->run( 'lint', @dirs );

Runs one command with its arguments (@ARGV as the shell gives it) and returns the exit code. help, --help and -h print the usage.

help

Prints the usage; returns 0.

cmd_lint

my $code = $cli->cmd_lint(@dirs);

holmgang lint: loads every suite (a suite directory, or each suite in a search directory) through "loader" -- structure, lint and both mock runs. Prints ok with every case id, or FAIL with the refusal (it names the case file and field; loading stops at the first refused case). Returns 1 when a suite was refused, 2 for a path that holds no suite. --seed ID=N (repeatable) replaces the seed of every generate block of the DSL suite ID ("seeds" in Langertha::Holmgang::Loader); an id that is no DSL suite of the command is a usage error.

cmd_export_yaml

my $code = $cli->cmd_export_yaml( @dirs, '--out', $root );

holmgang export-yaml: writes every suite in the case DSL (a suite directory with suite.pl, or each one in a search directory) as suite format 2 into --out/id ("export" in Langertha::Holmgang::Loader): it is loaded first, lint included, and the export must load back with the same digest. A format 2 suite among them is skipped (skip). Prints ok with the case count and the written directory, or FAIL with the refusal. --seed ID=N as for lint. Returns 1 when a suite was refused; 2 for a usage error -- no --out, no DSL suite, or a suite directory that already exists under --out (nothing is overwritten; checked before anything is written).

cmd_run

my $code = $cli->cmd_run( '--suite', $dir, '--target', 'mock-solution', '--report', $file );

holmgang run: loads the suites (--seed ID=N as for lint), builds the target (an engine name with --model, --url, --option; or mock-solution / mock-text) and its variants (--variant), and plays them through Langertha::Holmgang::Batch: every case --repeat times (default the suite's repeat, else 1) under every variant, at most --concurrency requests at once (default 1, one engine per slot), continuing the report of --resume. It prints a line per result as it ends (prefixed with the variant when there are several) and the evaluation, and writes the report with --report -- at the end, and as a checkpoint while the batch runs ("checkpoint_seconds"). Before the cases of a suite with images the batch runs the vision pre-check once per variant ("vision_check_f" in Langertha::Holmgang::Target); the outcome is printed; when it finds the target blind, every case with images of that suite ends no_vision without a request. SIGINT and SIGTERM cancel the batch: no further case starts, the cases in play finish, the report is written. Returns 1 when a verdict is not ok, 2 for a usage or setup error, an aborted or a cancelled run.

When the target answered no case of the whole run -- every result, over all suites and repetitions, is error, context_exceeded or no_vision -- the run measured nothing: it says so on the error handle with the count per status and the first error text of the run (the first result in suite, case and repetition order; already scrubbed by the target, put on one line, cut to 200 characters with an ellipsis) and returns 1, whatever the verdicts. The report is written as always. When at least one result was measured, the verdicts decide; a suite in which no case got an answer then gets a note on the error handle, and one whose results are all error also fails its verdict (NOT MEASURED, "verdicts" in Langertha::Holmgang::Evaluation), so the run returns 1; all context_exceeded or no_vision is a measurement and gets only the note. An aborted run stays 2. The engine's warnings while it is built go to the error handle, cleaned like a relayed error.

cmd_report

my $code = $cli->cmd_report( @files, '--format', 'markdown', '--suite', $dir );

holmgang report: reads the report files, joins them in one Langertha::Holmgang::Evaluation and prints it as text, Markdown or JSON. Thresholds come from the suites given with --suite; without them a verdict is judged by its hard cases, an unfinished run and whether anything was measured ("verdicts" in Langertha::Holmgang::Evaluation). Returns 1 when a verdict is not ok -- among them a suite whose results are all error, so a report of a run that measured nothing returns 1 as run did -- and 2 when a file does not read or the reports conflict.

SEE ALSO

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.