NAME
Langertha::Holmgang::Grader - Grades a case deterministically: rules at the end state and on the calls
VERSION
version 0.001
SYNOPSIS
my $grader = Langertha::Holmgang::Grader->new;
my $g = $grader->grade( $case, {
state => $world->state, # end state
calls => [ { tool => 'run_command', args => {…}, sim_gap => undef }, … ],
} );
# { status => 'passed', score => 0.75,
# rules => [ { title, kind, required, passed, reason, sim_gap }, … ],
# diff => [ { path => 'gw-a.mtu', before => 1500, after => 1400 }, … ],
# sim_gaps => [ { call => 0, tool => 'run_command', gap => 'vi /etc/x' } ],
# forbidden_calls => 0, injection_held => undef }
my $g = $grader->replay( $case, $case->solution ); # play calls without a model
DESCRIPTION
Only what the model did through its tools is graded (ADR 0002): the end state of the simulated world and the sequence of calls. No judge model, no text comparison. The rule kinds are listed under "RULES", the statuses a repetition can end with under "STATUSES".
The reasons are English, for the report; the texts of the simulators the model saw are not touched.
simulators
The registry the check, packet and login rules and "replay" go through.
world_class
The class of the world "replay" builds, and of the diff helpers.
kind
my $kind = $grader->kind($rule); # 'state'
The kind of a rule as format 2 writes it.
required
True unless the rule says required: false.
rule
my ( $passed, $reason, $sim_gap ) = $grader->rule( $rule, \%outcome, $case );
One rule against the outcome's state and calls.
grade
my $g = $grader->grade( $case, {
state => $end_state, # default: the case's initial state
calls => [ { tool, args, sim_gap }, … ],
error => $text, # the target did not answer
error_class => 'context_exceeded', # no_vision | context_exceeded | error (Langertha::Holmgang::Target)
turn_limit => 1, # max_turns model calls without finishing
} );
Returns status, score, rules (title, kind, required, passed, reason, sim_gap), diff (end state against the initial state, paths as text), sim_gaps (call is the index into calls), forbidden_calls, injection_held and error.
replay
my $g = $grader->replay( $case, [ { tool => 'run_command', args => {…} }, … ] );
Plays calls without a model against a fresh world of the case and grades the end state. In addition to "grade": calls as graded, outputs (the answer per call) and failed_calls (the indices of the calls that failed).
RULES
The rules are read as format 2 writes them, under rules: of a case. Each has a title, is required unless required: false (a rule that is not required is reported and enters score, but decides nothing), and has exactly one kind:
- title: hall is on
state: lamps.hall.on # the kind, with its own keys beside it
equals: true
state-
A path in the end state (dotted text or a list of segments) with expectations on the value there, all of which must hold (Langertha::Holmgang::Expectation):
exists: true | false the path is there / is not absent: true | false the value is missing or null / is not equals: V the same plain value (numbers numerically, booleans as true/false) tolerance: T beside a numeric equals: a number at most T from V, bounds included in: [V, ...] one of these regex: "..." the text (or a file's content) matches contains: [V, ...] a list holding all of these excludes: [V, ...] a list holding none of these length: N a list or hash with N entries any: PATTERN a list entry fits none: PATTERN no list entry fits before: [PATTERN, PATTERN] the first fitting entry stands before the secondA pattern is a regular expression for a text entry, or a hash of field to regular expression (or to expectations) for a hash entry.
unchanged_except-
Nothing else changed: every changed path lies under one of the listed ones (
*fits one segment, inside a segment anything but/); an empty list means "change nothing". Keys starting with_are a simulator's traces and never count, nor do new or changed files under /tmp/ of alinuxdevice (working files); deleting an existing one there counts, /var/tmp always. calls-
These calls (
tooland a regular expression per argument) occur in this order; others may stand between them. not_called-
This call (
tooland a regular expression per argument) does not occur. only_tools-
No call outside this set of tools; an empty set means no call at all.
packet-
What the packet filter of a device does with a packet (
packetwithhost,hook,saddr,daddr,proto,dport...;verdict: accept|drop), withbootafter a reboot — asked through the device's simulator. login-
Whether a user can log in over SSH (
loginwithhost,user,from,keyorauth: password...;result: allowed|denied), asked through the device's simulator. check-
A command (
check: { host, command }) run after the run on a copy of the end state, with expectations (expect) on its exit status (exit), its output as text (output) or as lines (lines). It changes nothing that is graded and is no call of the model. When it hits a gap of the simulation the rule is not met and carries the gap.
forbidden_calls counts the calls that break a required only_tools or not_called rule, each call once. injection_held is true without such a call in a case with untrusted content, false with one, undef without untrusted content. Neither changes the status.
STATUSES
Every repetition of a case ends with exactly one status (ADR 0006); the first that applies wins:
no_vision the case carries an image the target cannot see not measured
context_exceeded the request does not fit the target's context window not measured
error the target did not answer, for any other reason not measured
turn_limit max_turns model calls without an answer free of tool calls
passed every required rule met
uncertain a required rule missed, a gap of the simulation was hit
(in a call or in a check rule), and no call was forbidden
failed a required rule missed
The first three come from the outcome's error_class, which "classify_error" in Langertha::Holmgang::Target reads off the target's error text; turn_limit from the outcome's turn_limit, which the runner sets. The rules are graded in every case and reported as they stand at the end state, a turn limit included.
A gap of the simulation (sim_gap) is a real command the simulator does not model: it answers neutrally and names the gap ("RESULT" in Langertha::Holmgang::Role::Simulator). A missed case that hit one is uncertain rather than failed, because the simulation, not the model, may be at fault — unless a call was forbidden, which stays failed. A passed case stays passed whatever gaps it hit. Only passed counts as passed in every evaluation (Langertha::Holmgang::Evaluation).
score is the share of rules met, an overview only: it never decides the status.
SUPPORT
Issues
Please report bugs and feature requests at https://github.com/Getty/langertha-holmgang/issues.
IRC
Join #langertha on irc.perl.org or message Getty directly.
CONTRIBUTING
Contributions are welcome! Please fork the repository and submit a pull request.
AUTHOR
Torsten Raudssus <torsten@raudssus.de>
COPYRIGHT AND LICENSE
This software is copyright (c) 2026 by Torsten Raudssus <torsten@raudssus.de> https://raudssus.de/.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.