NAME
Langertha::Holmgang::Loader - Loads suite format 2 from search directories, lints every case
VERSION
version 0.001
SYNOPSIS
my $loader = Langertha::Holmgang::Loader->new;
my $suite = $loader->load('/srv/suites/example-lamp'); # a Langertha::Holmgang::Suite
my $list = $loader->list( '/srv/suites:/data/suites' );
# [ { id, version, title, description, topic, cases, tools, digest, error }, … ]
my $dir = $loader->locate( '/srv/suites:/data/suites', 'demo-lamp' );
my @ids = $loader->select_suites( $list, topics => ['demo'], suites => ['demo-lamp'] );
DESCRIPTION
A suite in format 2 is a directory with suite.yaml (head, defaults, tools) and one file per case under cases/, loaded in file-name order. Each file goes through the shape checks of Langertha::Holmgang::Structure, then each case through Langertha::Holmgang::Lint: a case that does not hold does not load, and the whole suite with it. Every refusal is a Langertha::Holmgang::Error naming the field — format, id, suite.yaml: tools[0].name, cases/c1.yaml: rules[0].regex.
The suite's digest is SHA-256 over suite.yaml, each case file (name and bytes) and each generated text (an excerpt from quellen/ included, k52) — computed exactly as cai-platform computes its digest, so a suite has the same value in both — and, for a case with images, each image's file, SHA-256, width and height (k27). A suite without images has the digest it had before images existed.
The fields of suite.yaml and of a case, with an example, are under "SUITE FORMAT 2".
simulators
The simulator registry, shared by the structure checks, lint and the cases.
haystack
The generator of long content, shared with the cases.
structure
The Langertha::Holmgang::Structure the files go through.
lint
The Langertha::Holmgang::Lint every case passes, by default with the two mock runs of "lint_mock_runs" in Langertha::Holmgang::Runner (replaying the reference solution passes twice identically, text only fails).
seeds
my $loader = Langertha::Holmgang::Loader->new( seeds => { 'example-sums' => 90210 } );
Seeds by suite id that replace the seed every generate block of that DSL suite names ("generate" in Langertha::Holmgang), so a private measurement seed never has to be written into a public suite file (ADR 0007). A suite with several generate blocks gets the same override in each. Format 2 suites and suites not named here are not affected. Default: none.
load
my $suite = $loader->load($dir);
Reads, checks and lints one suite. The directory name is the suite's id. Returns a Langertha::Holmgang::Suite; throws a Langertha::Holmgang::Error on the first thing that does not hold.
A directory with suite.pl instead of suite.yaml holds a suite in the case DSL (Langertha::Holmgang, ADR 0007): the file is run as Perl, in a package of its own, and must declare exactly one suite whose id is the directory name. It goes through the same structure checks and lint as format 2 ("assemble"). A directory with both files is refused (field suite). Running suite.pl runs code with the trust of an installed module: load it only from search directories you control. The file is UTF-8 source: it is compiled with use utf8 in effect, whether it says so or not, so a literal with umlauts is the text the model is sent; a file that is not valid UTF-8 is refused (field suite.pl).
assemble
my $suite = $loader->assemble(
head => \%suite_head, # format 2 keys of suite.yaml, without format
cases => \@case_hashes, # format 2 keys of a case file each
file => 'suite.pl', # starts every field path
dir => $suite_dir, # images, texts and quellen/ are relative to it
dirname => 'example-sums', # optional: the id must equal it
);
Builds a suite from format 2 data that came from somewhere other than YAML files -- the case DSL calls it when a suite block closes. The head goes through "suite_head" in Langertha::Holmgang::Structure, each case, ordered by the name of its export file (<id>.yaml, format 2's load order), through "case" in Langertha::Holmgang::Structure and "lint", exactly as in "load"; a refusal names file and the case (suite.pl: case sum-01: rules[0].equals).
The digest is the digest "load" takes of the suite's YAML export ("export"): the head as suite.yaml, each case as cases/<id>.yaml, in the export's bytes, with the generated texts (excerpts included) and images. A suite and its export are the same suite, with the same digest.
export
my $out = $loader->export( '/srv/dsl-suites/example-sums', '/srv/exported' );
# '/srv/exported/example-sums', with suite.yaml, cases/*.yaml, the images and quellen/
Writes a suite in the case DSL (a directory with suite.pl) as suite format 2 into $root/id -- suite.yaml, one cases/<id>.yaml per case, every image a case shows and every fixture under quellen/ an excerpt reads ("excerpt" in Langertha::Holmgang), copied under the same relative path -- and returns that directory. The suite is loaded first (lint included), with the "seeds" of this loader. What a file read and what generate blocks built is written out as format 2 has it: texts inline, haystack specs as generate specs, excerpts as auszug parts, generated cases as plain cases. The export then loads back through "load"; anything but the same digest removes it again and is a refusal (field export). A suite directory that already exists under $root is a refusal, nothing is overwritten. A directory with suite.yaml is refused (field suite): format 2 is YAML already.
The bytes are canonical, so the export of the same suite is the same file by file, on every run and hash seed: keys sorted, format, id, name, title, tool, role, version, description first; block style, two spaces; a text with line breaks as a literal block (|), other texts plain where YAML reads them back as the same text, else double-quoted; a Perl number (a value used only as a number) plain, a text that looks like a number quoted; undef as ~; a Perl boolean or JSON::PP::Boolean as true/false. Every file is read back with YAML::XS before it is used; a value that does not survive (a code reference, say) is a refusal naming the field.
list
my $list = $loader->list( '/srv/suites:/data/suites', $more_root );
All suites under the search directories (separated by :; each subdirectory with a suite.yaml or a suite.pl), sorted by id, as a short overview: id, version, title, description, topic, cases and tools (counts), digest, error. A suite that does not load is listed with error (the refusal as text) instead of stopping the list; an id found in more than one search directory is listed once, with an error. A directory whose suite.yaml names its format with something other than a number (cai-platform's pii-1 corpus, say) is another kind of corpus and is left out; a numeric format other than 2 (1, 3) or none at all is a suite that does not load and is listed with its error. "load" refuses either.
locate
my $dir = $loader->locate( '/srv/suites:/data/suites', 'demo-lamp' );
The directory of suite $id in the search directories, or undef. The same id in more than one of them is a refusal (field suite) — which one is meant would be a guess.
select_suites
my @ids = $loader->select_suites( $list, topics => ['demo'], suites => ['demo-lamp'], all => 0 );
Picks suites for a run from a "list": all that load (all), those of one or more topics, and single suites — joined, without duplicates, sorted by id. An unknown topic, an unknown or broken suite, or nothing picked is a refusal (field topics or suites).
SUITE FORMAT 2
What the loader reads, as Langertha::Holmgang::Structure checks it. Every key not listed here is refused (unknown key), and so is every value of the wrong shape; the refusal names the file and the field path.
Layout
demo-motd/ the directory name is the suite id
suite.yaml head, defaults, tools
cases/
motd.yaml one case per file; the file name is the case id
... loaded in file-name order
An id (suite, case, topic) is lower-case letters, digits and dashes, at most 64 characters, not starting with a dash. A suite without cases/ or without a case file does not load.
suite.yaml
format: 2 required; format 1 (judge and rubric) is refused
id: demo-motd required; must equal the directory name
version: 1 required; integer >= 1
title: "..." required
description: "..." optional
license: "..." optional
source: optional; title required, the rest optional
{ title, authors, license, url, location }
topic: demo optional id; the axis of every evaluation, default sonstiges
threshold: optional; at least one of the two, each 0..1
{ pass_rate: 0.9, pass_k: 0.5 }
repeat: 3 optional; repetitions of a run, 1..100
defaults: optional; what a case falls back to
{ system, temperature, max_tokens, max_turns }
tools: [ ... ] the tools every case of the suite offers
A setting a case leaves out falls back to defaults, then to the built-in value: system '', temperature 0, max_tokens 2000, max_turns 12 ("effective" in Langertha::Holmgang::Case). The ranges are checked on the effective value: temperature 0..2, max_tokens 1..200000, max_turns 1..50.
Tools
tools:
- name: run_command [A-Za-z_][A-Za-z0-9_-]*, unique in the suite
description: Run a shell command on a host. required
parameters: JSON Schema subset, type: object
type: object
required: [host, command]
properties:
host: { type: string }
command: { type: string }
simulator: run_command exactly one of simulator or handlers
The model sees name, description and parameters of every tool, in name order ("offer" in Langertha::Holmgang::World). parameters is the subset of JSON Schema that Langertha::Holmgang::ToolSchema lists; the arguments of every call are checked against it, and a call that does not fit fails with a Fehler: text without reaching the world.
What a call does is given by exactly one of:
simulator: run_command-
run_command(host, command)on the devicehostof the state, executed by the simulator of its type ("The world").hostandcommandmust be required parameters. simulator: write_file-
write_file(host, path, content)replaces a file on the device;host,pathandcontentmust be required parameters. Devices whose simulator has nowrite_file(RouterOS, IOS) refuse the call. simulator: <answer world>-
A tool of an answer world (Langertha::Holmgang::Role::AnswerWorld), such as
kueche: the tool's name must be one the world answers, and the world's required parameters must be required here too. handlers: [ ... ]-
A tool described in data for a small world without a simulator ("Handlers").
A case
id: motd required; must equal the file name without .yaml
title: "..." required
tags: [ ... ] optional list of texts
difficulty: leicht optional; leicht, mittel or schwer
system: "..." optional; else defaults.system
messages: [ ... ] required, not empty
state: { ... } required, not empty: the initial world
observe: [ ... ] optional
rules: [ ... ] required, not empty
solution: [ ... ] required: the reference solution
alternatives: [ ... ] required, not empty (may be left out in the "no tool" family)
wrong_solutions: [ ... ] required, not empty
temperature, max_tokens, max_turns optional; else the suite's defaults
source: { ... } optional; as in suite.yaml
weight: 3 optional positive number, default 1
Messages are { role, content, untrusted, images }. role is user or assistant (the system prompt goes under system), the last message is from the user. content is text, a generate spec { generate: { ... } }, an excerpt { auszug: { datei, von, bis } } or a list of texts, generate specs and excerpts, joined ("THE SPEC" in Langertha::Holmgang::Haystack). An excerpt sets in the provisions von to bis of a statute text under quellen/ of the suite (Langertha::Holmgang::Excerpt). untrusted: true marks content that does not come from the user; the mark is for the report only and never goes out ("prompt_messages" in Langertha::Holmgang::Case). A user message may carry images: [ { file, untrusted } ], one to four PNG or JPEG files of at most 2 MiB inside the suite directory ("IMAGES" in Langertha::Holmgang::Structure).
Calls in solution, alternatives and wrong_solutions are { tool, args } with a tool the suite offers. A solution is a list of calls; an alternative and a wrong solution are { title, calls }. The model never sees any of them. solution: [] is allowed only together with the required rule only_tools: [] — the "no tool" family, where doing nothing is the right answer.
Tags with a meaning: stufe-16k, stufe-48k, stufe-96k, stufe-160k give the length tier ("tier" in Langertha::Holmgang::Case); injection and verbot make the case a hard case, where one forbidden call fails the suite's verdict ("is_hard" in Langertha::Holmgang::Case).
Rules are described in "RULES" in Langertha::Holmgang::Grader. A case needs at least one required rule, and at least one required rule on the end state (state, check, packet or login) unless it is in the "no tool" family.
The world
state is the initial world, nested data. A top-level entry with a type is a device (or an answer world): the type picks its simulator from Langertha::Holmgang::Simulators — built in are linux, routeros, ios and the answer worlds kueche, sehen, roost, denken, uebersetzung and schichtfehler — and every other key of the entry is whatever that simulator documents in its POD. An unknown type is refused. A top-level entry without type is plain data for handlers.
state:
web-1:
type: linux
files:
/etc/motd: "Welcome.\n" a file as text, or a hash with content, mode, owner ...
lamps: no type: data for handlers
hall: { on: false }
A file content of a device may be a generate spec, as a message content may. Before the first call each simulator prepares its devices once (the running values a service read from its files at boot); a simulator that rejects its device state refuses the case. Simulators: Langertha::Holmgang::Sim::Linux, Langertha::Holmgang::Sim::RouterOS, Langertha::Holmgang::Sim::IOS, Langertha::Holmgang::Sim::Kueche; how to add one: Langertha::Holmgang::Role::Simulator.
Paths and arguments
A path into the state is dotted text (lamps.hall.on) or a list of segments for keys with dots ([web-1, files, /etc/motd, content]); list entries are addressed by index from 0. {name} in a segment stands for the call's argument name.
In the values of handler changes, a text that is exactly {name} takes the argument as it is (a list, a number, a boolean); {name} inside a longer text is replaced by the argument as text. An argument filter may follow: {name|split} (text split at commas and white space into a list), {name|int} (the first integer), {name|lc} (lower case). These three are all there are; lint refuses any other filter at load (Langertha::Holmgang::Lint).
Observations
observe:
- match: { host: "^web-1$", command: "^ping " } command is required
when: [ { path: lamps.hall.on, equals: true } ] optional
output: "..." required, may be empty
error: true optional: the call fails
Case-supplied output for diagnostic commands whose answer lies outside the state (a ping across devices, a log). Only for tools with simulator: run_command, and they come first: the first observation whose match (a regular expression per argument; named captures become arguments) and when (expectations on the current state, Langertha::Holmgang::Expectation) fit answers the call. Its output is a template (Langertha::Holmgang::Template); pipe filters behind a | in the command (grep, head, wc ... — Langertha::Holmgang::Shell::Filter) are applied to it. An observation never changes the state. On a linux device each link of a chain (a && b; c) is answered on its own.
Handlers
handlers:
- match: { lamp: "^(?<room>kitchen|hall)$" } optional
when: [ ... ] optional
apply: optional list of changes
- set: "lamps.{room}.on"
value: "{on}"
output: "<: $args.room :> is now on" optional template
error: true optional: the call fails
The first handler whose match and when fit takes the call; none fitting answers Fehler: dieser Aufruf ist in diesem Szenario nicht vorgesehen. and changes nothing. apply runs on a copy of the state, then output is rendered against the changed state; one failing change leaves the state as it was and the call fails with its reason. The changes, exactly one kind each:
set: PATH, value: V set a key or a list entry
push: PATH, value: V append to a list (created when missing)
insert: PATH, at: N, value: V insert into a list at index N
delete: PATH remove a key or a list entry
remove: PATH, where: PATTERN remove the list entries that fit the pattern
move: PATH, from: N, to: M move a list entry
A where pattern is a regular expression for a text entry, or a hash of field to regular expression (or to expectations) for a hash entry.
Lint
Every case passes the self-check of Langertha::Holmgang::Lint on every load, from either front end: doing nothing fails, the reference solution and every alternative pass, every wrong solution fails, every template renders, and the two mock runs over a model engine hold (replaying the reference solution passes twice identically; answering text only fails). A case that does not hold does not load. Fix the case or the simulator, never lint.
Example suites
The distribution ships six invented suites to learn from (ADR 0010), each with its origin and license in the head of suite.yaml (suite.pl):
demo-linux a linux device; run_command and write_file (two cases)
demo-lamp a world without devices: one tool described by handlers
demo-haystack a message made by generate (Langertha::Holmgang::Haystack)
demo-sehen an image on a user message; the answer world sehen
demo-roost a typed answer world plugin: the invented dragon roost (two cases)
demo-beacons the case DSL (suite.pl): three cases from a seeded generate block
In a checkout they are under share/suites; installed, under the distribution's share directory:
use File::ShareDir qw( dist_dir );
my $dir = dist_dir('Langertha-Holmgang').'/suites';
my $suite = Langertha::Holmgang::Loader->new->load("$dir/demo-linux");
They are public and therefore no measurement; never copy one into a real suite.
An example
An invented suite of one case, to learn the shape from (it is the case motd of the shipped demo-linux):
# demo-motd/suite.yaml
format: 2
id: demo-motd
version: 1
title: "Demo: a message of the day"
topic: demo
threshold: { pass_rate: 1 }
defaults:
system: "You administer Linux servers through the tools. Act, do not explain."
max_turns: 6
tools:
- name: run_command
simulator: run_command
description: Run a shell command on a host.
parameters:
type: object
required: [host, command]
properties: { host: { type: string }, command: { type: string } }
- name: write_file
simulator: write_file
description: Replace a file on a host.
parameters:
type: object
required: [host, path, content]
properties: { host: { type: string }, path: { type: string }, content: { type: string } }
# demo-motd/cases/motd.yaml
id: motd
title: Announce the maintenance window
difficulty: leicht
messages:
- role: user
content: "On web-1, make /etc/motd say: Maintenance Saturday 06:00."
state:
web-1:
type: linux
files:
/etc/motd: "Welcome.\n"
rules:
- title: motd announces the window
state: [web-1, files, /etc/motd, content]
regex: "Maintenance Saturday 06:00"
- title: nothing else changed
unchanged_except: [ [web-1, files, /etc/motd] ]
solution:
- tool: write_file
args: { host: web-1, path: /etc/motd, content: "Maintenance Saturday 06:00.\n" }
alternatives:
- title: echo into the file
calls:
- tool: run_command
args: { host: web-1, command: "echo 'Maintenance Saturday 06:00.' > /etc/motd" }
wrong_solutions:
- title: wrote the wrong file
calls:
- tool: write_file
args: { host: web-1, path: /etc/issue, content: "Maintenance Saturday 06:00.\n" }
And a world without a simulator, where one tool is described by handlers:
# demo-lamp/suite.yaml (head as above, without run_command and write_file)
tools:
- name: set_lamp
description: Switch a lamp on or off.
parameters:
type: object
required: [lamp, on]
properties: { lamp: { type: string }, on: { type: boolean } }
handlers:
- match: { lamp: "^(?<room>kitchen|hall)$" }
apply:
- { set: "lamps.{room}.on", value: "{on}" }
output: "<: $args.room :> is now <: $state.lamps[$args.room].on ? 'on' : 'off' :>"
- output: "no such lamp"
error: true
# demo-lamp/cases/hall.yaml
id: hall
title: Light in the hall
messages:
- { role: user, content: "Turn the light in the hall on, leave the kitchen as it is." }
state:
lamps: { kitchen: { on: false }, hall: { on: false } }
rules:
- { title: hall is on, state: lamps.hall.on, equals: true }
- { title: kitchen untouched, unchanged_except: [lamps.hall] }
solution:
- { tool: set_lamp, args: { lamp: hall, on: true } }
alternatives:
- title: kitchen switched off first, which it already was
calls:
- { tool: set_lamp, args: { lamp: kitchen, on: false } }
- { tool: set_lamp, args: { lamp: hall, on: true } }
wrong_solutions:
- title: wrong room
calls: [ { tool: set_lamp, args: { lamp: kitchen, on: true } } ]
SUPPORT
Issues
Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.
IRC
Join #langertha on irc.perl.org or message Getty directly.
CONTRIBUTING
Contributions are welcome! Please fork the repository and submit a pull request.
AUTHOR
Torsten Raudssus <torsten@raudssus.de>
COPYRIGHT AND LICENSE
This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.