NAME

Langertha::Holmgang::Loader - Loads suite format 2 from search directories, lints every case

VERSION

version 0.001

SYNOPSIS

my $loader = Langertha::Holmgang::Loader->new;

my $suite = $loader->load('/srv/suites/example-lamp');   # a Langertha::Holmgang::Suite

my $list = $loader->list( '/srv/suites:/data/suites' );
# [ { id, version, title, description, topic, cases, tools, digest, error }, … ]

my $dir = $loader->locate( '/srv/suites:/data/suites', 'demo-lamp' );
my @ids = $loader->select_suites( $list, topics => ['demo'], suites => ['demo-lamp'] );

DESCRIPTION

A suite in format 2 is a directory with suite.yaml (head, defaults, tools) and one file per case under cases/, loaded in file-name order. Each file goes through the shape checks of Langertha::Holmgang::Structure, then each case through Langertha::Holmgang::Lint: a case that does not hold does not load, and the whole suite with it. Every refusal is a Langertha::Holmgang::Error naming the field — format, id, suite.yaml: tools[0].name, cases/c1.yaml: rules[0].regex.

The suite's digest is SHA-256 over suite.yaml, each case file (name and bytes) and each generated text (an excerpt from quellen/ included, k52) — computed exactly as cai-platform computes its digest, so a suite has the same value in both — and, for a case with images, each image's file, SHA-256, width and height (k27). A suite without images has the digest it had before images existed.

The fields of suite.yaml and of a case, with an example, are under "SUITE FORMAT 2".

simulators

The simulator registry, shared by the structure checks, lint and the cases.

haystack

The generator of long content, shared with the cases.

structure

The Langertha::Holmgang::Structure the files go through.

lint

The Langertha::Holmgang::Lint every case passes, by default with the two mock runs of "lint_mock_runs" in Langertha::Holmgang::Runner (replaying the reference solution passes twice identically, text only fails).

seeds

my $loader = Langertha::Holmgang::Loader->new( seeds => { 'example-sums' => 90210 } );

Seeds by suite id that replace the seed every generate block of that DSL suite names ("generate" in Langertha::Holmgang), so a private measurement seed never has to be written into a public suite file (ADR 0007). A suite with several generate blocks gets the same override in each. Format 2 suites and suites not named here are not affected. Default: none.

load

my $suite = $loader->load($dir);

Reads, checks and lints one suite. The directory name is the suite's id. Returns a Langertha::Holmgang::Suite; throws a Langertha::Holmgang::Error on the first thing that does not hold.

A directory with suite.pl instead of suite.yaml holds a suite in the case DSL (Langertha::Holmgang, ADR 0007): the file is run as Perl, in a package of its own, and must declare exactly one suite whose id is the directory name. It goes through the same structure checks and lint as format 2 ("assemble"). A directory with both files is refused (field suite). Running suite.pl runs code with the trust of an installed module: load it only from search directories you control. The file is UTF-8 source: it is compiled with use utf8 in effect, whether it says so or not, so a literal with umlauts is the text the model is sent; a file that is not valid UTF-8 is refused (field suite.pl).

assemble

my $suite = $loader->assemble(
  head    => \%suite_head,    # format 2 keys of suite.yaml, without format
  cases   => \@case_hashes,   # format 2 keys of a case file each
  file    => 'suite.pl',      # starts every field path
  dir     => $suite_dir,      # images, texts and quellen/ are relative to it
  dirname => 'example-sums',  # optional: the id must equal it
);

Builds a suite from format 2 data that came from somewhere other than YAML files -- the case DSL calls it when a suite block closes. The head goes through "suite_head" in Langertha::Holmgang::Structure, each case, ordered by the name of its export file (<id>.yaml, format 2's load order), through "case" in Langertha::Holmgang::Structure and "lint", exactly as in "load"; a refusal names file and the case (suite.pl: case sum-01: rules[0].equals).

The digest is the digest "load" takes of the suite's YAML export ("export"): the head as suite.yaml, each case as cases/<id>.yaml, in the export's bytes, with the generated texts (excerpts included) and images. A suite and its export are the same suite, with the same digest.

export

my $out = $loader->export( '/srv/dsl-suites/example-sums', '/srv/exported' );
# '/srv/exported/example-sums', with suite.yaml, cases/*.yaml, the images and quellen/

Writes a suite in the case DSL (a directory with suite.pl) as suite format 2 into $root/id -- suite.yaml, one cases/<id>.yaml per case, every image a case shows and every fixture under quellen/ an excerpt reads ("excerpt" in Langertha::Holmgang), copied under the same relative path -- and returns that directory. The suite is loaded first (lint included), with the "seeds" of this loader. What a file read and what generate blocks built is written out as format 2 has it: texts inline, haystack specs as generate specs, excerpts as auszug parts, generated cases as plain cases. The export then loads back through "load"; anything but the same digest removes it again and is a refusal (field export). A suite directory that already exists under $root is a refusal, nothing is overwritten. A directory with suite.yaml is refused (field suite): format 2 is YAML already.

The bytes are canonical, so the export of the same suite is the same file by file, on every run and hash seed: keys sorted, format, id, name, title, tool, role, version, description first; block style, two spaces; a text with line breaks as a literal block (|), other texts plain where YAML reads them back as the same text, else double-quoted; a Perl number (a value used only as a number) plain, a text that looks like a number quoted; undef as ~; a Perl boolean or JSON::PP::Boolean as true/false. Every file is read back with YAML::XS before it is used; a value that does not survive (a code reference, say) is a refusal naming the field.

list

my $list = $loader->list( '/srv/suites:/data/suites', $more_root );

All suites under the search directories (separated by :; each subdirectory with a suite.yaml or a suite.pl), sorted by id, as a short overview: id, version, title, description, topic, cases and tools (counts), digest, error. A suite that does not load is listed with error (the refusal as text) instead of stopping the list; an id found in more than one search directory is listed once, with an error. A directory whose suite.yaml names its format with something other than a number (cai-platform's pii-1 corpus, say) is another kind of corpus and is left out; a numeric format other than 2 (1, 3) or none at all is a suite that does not load and is listed with its error. "load" refuses either.

locate

my $dir = $loader->locate( '/srv/suites:/data/suites', 'demo-lamp' );

The directory of suite $id in the search directories, or undef. The same id in more than one of them is a refusal (field suite) — which one is meant would be a guess.

select_suites

my @ids = $loader->select_suites( $list, topics => ['demo'], suites => ['demo-lamp'], all => 0 );

Picks suites for a run from a "list": all that load (all), those of one or more topics, and single suites — joined, without duplicates, sorted by id. An unknown topic, an unknown or broken suite, or nothing picked is a refusal (field topics or suites).

SUITE FORMAT 2

What the loader reads, as Langertha::Holmgang::Structure checks it. Every key not listed here is refused (unknown key), and so is every value of the wrong shape; the refusal names the file and the field path.

Layout

demo-motd/              the directory name is the suite id
  suite.yaml            head, defaults, tools
  cases/
    motd.yaml           one case per file; the file name is the case id
    ...                 loaded in file-name order

An id (suite, case, topic) is lower-case letters, digits and dashes, at most 64 characters, not starting with a dash. A suite without cases/ or without a case file does not load.

suite.yaml

format: 2                  required; format 1 (judge and rubric) is refused
id: demo-motd              required; must equal the directory name
version: 1                 required; integer >= 1
title: "..."               required
description: "..."         optional
license: "..."             optional
source:                    optional; title required, the rest optional
  { title, authors, license, url, location }
topic: demo                optional id; the axis of every evaluation, default sonstiges
threshold:                 optional; at least one of the two, each 0..1
  { pass_rate: 0.9, pass_k: 0.5 }
repeat: 3                  optional; repetitions of a run, 1..100
defaults:                  optional; what a case falls back to
  { system, temperature, max_tokens, max_turns }
tools: [ ... ]             the tools every case of the suite offers

A setting a case leaves out falls back to defaults, then to the built-in value: system '', temperature 0, max_tokens 2000, max_turns 12 ("effective" in Langertha::Holmgang::Case). The ranges are checked on the effective value: temperature 0..2, max_tokens 1..200000, max_turns 1..50.

Tools

tools:
  - name: run_command          [A-Za-z_][A-Za-z0-9_-]*, unique in the suite
    description: Run a shell command on a host.     required
    parameters:                JSON Schema subset, type: object
      type: object
      required: [host, command]
      properties:
        host:    { type: string }
        command: { type: string }
    simulator: run_command     exactly one of simulator or handlers

The model sees name, description and parameters of every tool, in name order ("offer" in Langertha::Holmgang::World). parameters is the subset of JSON Schema that Langertha::Holmgang::ToolSchema lists; the arguments of every call are checked against it, and a call that does not fit fails with a Fehler: text without reaching the world.

What a call does is given by exactly one of:

simulator: run_command

run_command(host, command) on the device host of the state, executed by the simulator of its type ("The world"). host and command must be required parameters.

simulator: write_file

write_file(host, path, content) replaces a file on the device; host, path and content must be required parameters. Devices whose simulator has no write_file (RouterOS, IOS) refuse the call.

simulator: <answer world>

A tool of an answer world (Langertha::Holmgang::Role::AnswerWorld), such as kueche: the tool's name must be one the world answers, and the world's required parameters must be required here too.

handlers: [ ... ]

A tool described in data for a small world without a simulator ("Handlers").

A case

id: motd                   required; must equal the file name without .yaml
title: "..."               required
tags: [ ... ]              optional list of texts
difficulty: leicht         optional; leicht, mittel or schwer
system: "..."              optional; else defaults.system
messages: [ ... ]          required, not empty
state: { ... }             required, not empty: the initial world
observe: [ ... ]           optional
rules: [ ... ]             required, not empty
solution: [ ... ]          required: the reference solution
alternatives: [ ... ]      required, not empty (may be left out in the "no tool" family)
wrong_solutions: [ ... ]   required, not empty
temperature, max_tokens, max_turns     optional; else the suite's defaults
source: { ... }            optional; as in suite.yaml
weight: 3                  optional positive number, default 1

Messages are { role, content, untrusted, images }. role is user or assistant (the system prompt goes under system), the last message is from the user. content is text, a generate spec { generate: { ... } }, an excerpt { auszug: { datei, von, bis } } or a list of texts, generate specs and excerpts, joined ("THE SPEC" in Langertha::Holmgang::Haystack). An excerpt sets in the provisions von to bis of a statute text under quellen/ of the suite (Langertha::Holmgang::Excerpt). untrusted: true marks content that does not come from the user; the mark is for the report only and never goes out ("prompt_messages" in Langertha::Holmgang::Case). A user message may carry images: [ { file, untrusted } ], one to four PNG or JPEG files of at most 2 MiB inside the suite directory ("IMAGES" in Langertha::Holmgang::Structure).

Calls in solution, alternatives and wrong_solutions are { tool, args } with a tool the suite offers. A solution is a list of calls; an alternative and a wrong solution are { title, calls }. The model never sees any of them. solution: [] is allowed only together with the required rule only_tools: [] — the "no tool" family, where doing nothing is the right answer.

Tags with a meaning: stufe-16k, stufe-48k, stufe-96k, stufe-160k give the length tier ("tier" in Langertha::Holmgang::Case); injection and verbot make the case a hard case, where one forbidden call fails the suite's verdict ("is_hard" in Langertha::Holmgang::Case).

Rules are described in "RULES" in Langertha::Holmgang::Grader. A case needs at least one required rule, and at least one required rule on the end state (state, check, packet or login) unless it is in the "no tool" family.

The world

state is the initial world, nested data. A top-level entry with a type is a device (or an answer world): the type picks its simulator from Langertha::Holmgang::Simulators — built in are linux, routeros, ios and the answer worlds kueche, sehen, roost, denken, uebersetzung and schichtfehler — and every other key of the entry is whatever that simulator documents in its POD. An unknown type is refused. A top-level entry without type is plain data for handlers.

state:
  web-1:
    type: linux
    files:
      /etc/motd: "Welcome.\n"            a file as text, or a hash with content, mode, owner ...
  lamps:                                 no type: data for handlers
    hall: { on: false }

A file content of a device may be a generate spec, as a message content may. Before the first call each simulator prepares its devices once (the running values a service read from its files at boot); a simulator that rejects its device state refuses the case. Simulators: Langertha::Holmgang::Sim::Linux, Langertha::Holmgang::Sim::RouterOS, Langertha::Holmgang::Sim::IOS, Langertha::Holmgang::Sim::Kueche; how to add one: Langertha::Holmgang::Role::Simulator.

Paths and arguments

A path into the state is dotted text (lamps.hall.on) or a list of segments for keys with dots ([web-1, files, /etc/motd, content]); list entries are addressed by index from 0. {name} in a segment stands for the call's argument name.

In the values of handler changes, a text that is exactly {name} takes the argument as it is (a list, a number, a boolean); {name} inside a longer text is replaced by the argument as text. An argument filter may follow: {name|split} (text split at commas and white space into a list), {name|int} (the first integer), {name|lc} (lower case). These three are all there are; lint refuses any other filter at load (Langertha::Holmgang::Lint).

Observations

observe:
  - match: { host: "^web-1$", command: "^ping " }     command is required
    when:  [ { path: lamps.hall.on, equals: true } ]  optional
    output: "..."                                     required, may be empty
    error: true                                       optional: the call fails

Case-supplied output for diagnostic commands whose answer lies outside the state (a ping across devices, a log). Only for tools with simulator: run_command, and they come first: the first observation whose match (a regular expression per argument; named captures become arguments) and when (expectations on the current state, Langertha::Holmgang::Expectation) fit answers the call. Its output is a template (Langertha::Holmgang::Template); pipe filters behind a | in the command (grep, head, wc ... — Langertha::Holmgang::Shell::Filter) are applied to it. An observation never changes the state. On a linux device each link of a chain (a && b; c) is answered on its own.

Handlers

handlers:
  - match: { lamp: "^(?<room>kitchen|hall)$" }        optional
    when:  [ ... ]                                    optional
    apply:                                            optional list of changes
      - set: "lamps.{room}.on"
        value: "{on}"
    output: "<: $args.room :> is now on"              optional template
    error: true                                       optional: the call fails

The first handler whose match and when fit takes the call; none fitting answers Fehler: dieser Aufruf ist in diesem Szenario nicht vorgesehen. and changes nothing. apply runs on a copy of the state, then output is rendered against the changed state; one failing change leaves the state as it was and the call fails with its reason. The changes, exactly one kind each:

set: PATH, value: V              set a key or a list entry
push: PATH, value: V             append to a list (created when missing)
insert: PATH, at: N, value: V    insert into a list at index N
delete: PATH                     remove a key or a list entry
remove: PATH, where: PATTERN     remove the list entries that fit the pattern
move: PATH, from: N, to: M       move a list entry

A where pattern is a regular expression for a text entry, or a hash of field to regular expression (or to expectations) for a hash entry.

Lint

Every case passes the self-check of Langertha::Holmgang::Lint on every load, from either front end: doing nothing fails, the reference solution and every alternative pass, every wrong solution fails, every template renders, and the two mock runs over a model engine hold (replaying the reference solution passes twice identically; answering text only fails). A case that does not hold does not load. Fix the case or the simulator, never lint.

Example suites

The distribution ships six invented suites to learn from (ADR 0010), each with its origin and license in the head of suite.yaml (suite.pl):

demo-linux      a linux device; run_command and write_file (two cases)
demo-lamp       a world without devices: one tool described by handlers
demo-haystack   a message made by generate (Langertha::Holmgang::Haystack)
demo-sehen      an image on a user message; the answer world sehen
demo-roost      a typed answer world plugin: the invented dragon roost (two cases)
demo-beacons    the case DSL (suite.pl): three cases from a seeded generate block

In a checkout they are under share/suites; installed, under the distribution's share directory:

use File::ShareDir qw( dist_dir );
my $dir = dist_dir('Langertha-Holmgang').'/suites';
my $suite = Langertha::Holmgang::Loader->new->load("$dir/demo-linux");

They are public and therefore no measurement; never copy one into a real suite.

An example

An invented suite of one case, to learn the shape from (it is the case motd of the shipped demo-linux):

# demo-motd/suite.yaml
format: 2
id: demo-motd
version: 1
title: "Demo: a message of the day"
topic: demo
threshold: { pass_rate: 1 }
defaults:
  system: "You administer Linux servers through the tools. Act, do not explain."
  max_turns: 6
tools:
  - name: run_command
    simulator: run_command
    description: Run a shell command on a host.
    parameters:
      type: object
      required: [host, command]
      properties: { host: { type: string }, command: { type: string } }
  - name: write_file
    simulator: write_file
    description: Replace a file on a host.
    parameters:
      type: object
      required: [host, path, content]
      properties: { host: { type: string }, path: { type: string }, content: { type: string } }

# demo-motd/cases/motd.yaml
id: motd
title: Announce the maintenance window
difficulty: leicht
messages:
  - role: user
    content: "On web-1, make /etc/motd say: Maintenance Saturday 06:00."
state:
  web-1:
    type: linux
    files:
      /etc/motd: "Welcome.\n"
rules:
  - title: motd announces the window
    state: [web-1, files, /etc/motd, content]
    regex: "Maintenance Saturday 06:00"
  - title: nothing else changed
    unchanged_except: [ [web-1, files, /etc/motd] ]
solution:
  - tool: write_file
    args: { host: web-1, path: /etc/motd, content: "Maintenance Saturday 06:00.\n" }
alternatives:
  - title: echo into the file
    calls:
      - tool: run_command
        args: { host: web-1, command: "echo 'Maintenance Saturday 06:00.' > /etc/motd" }
wrong_solutions:
  - title: wrote the wrong file
    calls:
      - tool: write_file
        args: { host: web-1, path: /etc/issue, content: "Maintenance Saturday 06:00.\n" }

And a world without a simulator, where one tool is described by handlers:

# demo-lamp/suite.yaml (head as above, without run_command and write_file)
tools:
  - name: set_lamp
    description: Switch a lamp on or off.
    parameters:
      type: object
      required: [lamp, on]
      properties: { lamp: { type: string }, on: { type: boolean } }
    handlers:
      - match: { lamp: "^(?<room>kitchen|hall)$" }
        apply:
          - { set: "lamps.{room}.on", value: "{on}" }
        output: "<: $args.room :> is now <: $state.lamps[$args.room].on ? 'on' : 'off' :>"
      - output: "no such lamp"
        error: true

# demo-lamp/cases/hall.yaml
id: hall
title: Light in the hall
messages:
  - { role: user, content: "Turn the light in the hall on, leave the kitchen as it is." }
state:
  lamps: { kitchen: { on: false }, hall: { on: false } }
rules:
  - { title: hall is on, state: lamps.hall.on, equals: true }
  - { title: kitchen untouched, unchanged_except: [lamps.hall] }
solution:
  - { tool: set_lamp, args: { lamp: hall, on: true } }
alternatives:
  - title: kitchen switched off first, which it already was
    calls:
      - { tool: set_lamp, args: { lamp: kitchen, on: false } }
      - { tool: set_lamp, args: { lamp: hall, on: true } }
wrong_solutions:
  - title: wrong room
    calls: [ { tool: set_lamp, args: { lamp: kitchen, on: true } } ]

SUPPORT

Issues

Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.

IRC

Join #langertha on irc.perl.org or message Getty directly.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.