NAME

App::Test::Generator::BenchmarkGenerator - Generate Benchmark harnesses from ATG schemas

VERSION

Version 0.49

SYNOPSIS

Pattern 1: read a schema file and print the benchmark script.

use App::Test::Generator::BenchmarkGenerator;
use YAML::XS qw(LoadFile);

my $schema = LoadFile('schemas/my_func.yml');
my $bg     = App::Test::Generator::BenchmarkGenerator->new(schema => $schema);
print $bg->generate();

Pattern 2: write the schema by hand, for a plain function that takes its arguments in a fixed order, and save the script to a file.

my $bg = App::Test::Generator::BenchmarkGenerator->new(
    schema => {
        module   => 'List::Util',
        function => 'max',
        input    => {
            first  => { type => 'integer', position => 0, min => 1, max => 9 },
            second => { type => 'integer', position => 1 },
        },
    },
);

open my $fh, '>', 'benchmarks/max.pl' or die $!;
print $fh $bg->generate();    # the call is: List::Util::max(5, 42)
close $fh;

Pattern 3: compare several kinds of input for an object method. Each entry under transforms becomes one line in the timing table.

my $bg = App::Test::Generator::BenchmarkGenerator->new(
    schema => {
        module     => 'My::Shop',
        function   => 'price',
        new        => { currency => 'GBP' },
        input      => { amount => { type => 'number' } },
        transforms => {
            small => { input => { amount => { type => 'number', max => 10 } } },
            large => { input => { amount => { type => 'number', min => 1000 } } },
        },
    },
);
my $script = $bg->generate();
# The script builds:  my $obj = My::Shop->new(currency => 'GBP');
# and then times:     $obj->price(amount => 9)
#                     $obj->price(amount => 1001)

Pattern 4: benchmark a Perl built-in function. No use line is written for the special module name builtin.

my $bg = App::Test::Generator::BenchmarkGenerator->new(
    schema => {
        module   => 'builtin',
        function => 'abs',
        input    => { n => { type => 'number', position => 0 } },
    },
);
print $bg->generate();    # the call is: abs(42)

DESCRIPTION

Given an ATG YAML schema (as produced by extract-schemas or written by hand), generates a self-contained Perl benchmark script using "cmpthese" in Benchmark.

Each transform defined in the schema becomes one variant in the cmpthese call, with representative input values derived from the transform's type and range constraints. When no transforms are defined, a single 'default' variant is emitted using the base input specification.

The generated script is a plain .pl file suitable for running directly with perl. It is not a test file and has no dependency on any test framework.

METHODS

new

Construct a new BenchmarkGenerator for a given ATG schema.

my $bg = App::Test::Generator::BenchmarkGenerator->new(schema => $schema);

Arguments

  • schema

    A hashref representing the parsed YAML schema for the target function, as produced by extract-schemas or written by hand. Must contain at minimum module and function keys. Required.

Returns

A blessed App::Test::Generator::BenchmarkGenerator object. Croaks if schema is missing or not a hashref.

EXAMPLE

use YAML::XS qw(LoadFile);
use App::Test::Generator::BenchmarkGenerator;

my $schema = LoadFile('schemas/greet.yml');
my $bg     = App::Test::Generator::BenchmarkGenerator->new(schema => $schema);
print $bg->generate();

MESSAGES

schema is required

schema was not supplied or was undef.

schema must be a hashref

schema was supplied but is not a plain hashref (e.g. an arrayref or string was passed instead).

API SPECIFICATION

input

{
    schema => { type => 'hashref' },
}

output

{
    type => 'object',
    isa  => 'App::Test::Generator::BenchmarkGenerator',
}

The mathematical description is in "FORMAL SPECIFICATION".

generate

Generate the complete benchmark script as a string.

my $script = $bg->generate();
write_file('benchmarks/greet.pl', $script);

Arguments

None beyond $self.

Returns

A string containing the complete, self-contained Perl benchmark script using Benchmark::cmpthese. The string is ready to write directly to a .pl file and run with perl.

Croaks if the schema is missing a module or function key, or if either name is not a valid Perl name.

The object is not changed by this call. You can call generate many times and get the same text each time.

EXAMPLE

my $script = $bg->generate();
# Write to file
open my $fh, '>', 'benchmarks/my_func.pl' or die $!;
print $fh $script;
close $fh;

# Or run immediately:
require File::Temp;
my $tmp = File::Temp->new(SUFFIX => '.pl');
print $tmp $script;
system($^X, "$tmp");

MESSAGES

schema missing module

The schema hashref has no module key.

schema missing function

The schema hashref has no function key.

BenchmarkGenerator: module 'NAME' is not a valid Perl identifier

The module value is not a Perl package name such as My::Module. The special name builtin is always accepted.

BenchmarkGenerator: function 'NAME' is not a valid Perl identifier

The function value is not a Perl name such as greet or My::Module::greet.

BenchmarkGenerator: cannot write a TYPE reference as a literal

A value under new is a code reference, an object or another kind of reference that cannot be written as Perl source. Plain values, arrayrefs and hashrefs are accepted.

API SPECIFICATION

input

None (reads from the schema passed to new).

output

{ type => 'string' }

The string is the complete benchmark script, ready to write to a file.

The mathematical description is in "FORMAL SPECIFICATION".

PSEUDOCODE

read module, function, input, transforms from schema
emit shebang, use strict/warnings, use Benchmark
unless module eq 'builtin': emit use $module
if schema has 'new' key and not builtin:
    emit my $obj = Module->new(...)
if transforms defined and non-empty:
    for each transform: build a named variant call
else:
    build a single 'default' variant call
emit cmpthese($COUNT, { variant => sub { ... }, ... })

COMMON PITFALLS

Schema missing module or function

generate croaks immediately if either key is absent. Always ensure the schema has been loaded from a valid ATG YAML file before calling generate.

Expecting test-framework output

The generated script uses Benchmark::cmpthese and prints timing results to STDOUT. It is not a test file and produces no TAP output. Do not run it with prove.

OOP schemas need a new: key

If the function under benchmark is an instance method, the schema must have a new: key (even if its value is an empty hashref {}) so generate emits a my $obj = Module-new(...)> constructor call before the cmpthese block. Without it, the variant calls use Module::function(...) form.

Transforms that omit the input key

If a transform's hashref has no input key, generate falls back to the base schema input spec for that variant. This is intentional but may produce identical argument lists across variants if you forgot to add per- transform input overrides.

A transform input replaces the base input. It is not merged.

If the base input has the parameters a and b, and a transform input has only a, then that variant is called with a alone. b is not copied from the base. Repeat every parameter you need inside each transform.

Only plain data is allowed under new

A value under new can be a number, a string, undef, an arrayref or a hashref, nested to any depth. A code reference or an object makes generate croak.

How undef is handled
  • schema => undef makes new croak with schema is required.

  • A module or function that is undef makes generate croak.

  • A missing or undef input or transforms key means "no parameters" or "no transforms". This is safe.

  • new => undef (new: ~ in YAML) still counts as "has a constructor". The script calls Module->new() with no arguments.

  • An undef value inside new is written as the Perl word undef.

  • A parameter whose specification is undef is passed as the Perl word undef.

The text undef means "no input"

Some ATG schema files contain input: undef. YAML reads this as the five-letter string "undef". generate treats that string, and any other input or transforms value that is not a hashref, as "no parameters". Inside a transform, input: undef gives a call with no arguments; it does not fall back to the base input.

Names that are not plain words are quoted for you

Transform names, parameter names and the keys under new can contain any characters. They are quoted and escaped in the generated script when needed. Only module and function must be valid Perl names.

Which text is written as a number

A string under new is written without quotes only when it is a plain decimal number such as 42, -1.5 or 2e10. Everything else, including 'Inf', 'NaN' and ' 12', is written as a quoted string. Be careful with a leading zero: '007' is written as 007, which Perl reads as an octal number.

ENCODING

  • module and function

    Use ASCII letters, digits, underscores and :: only. Non-ASCII letters may pass the name check, but the generated script has no use utf8 line, so Perl will not read such names correctly.

  • Transform names, parameter names and string values under new

    Full UTF-8 text, including non-ASCII letters and emojis, is copied into the script unchanged. This is safe only if you take two steps yourself: write the script through a UTF-8 file handle (open my $fh, '>:encoding(UTF-8)', $path), and remember that the script has no use utf8 line, so the running benchmark sees the text as bytes, not as characters. For a benchmark that needs real character strings, add use utf8; to the generated file by hand.

  • Sample argument values

    These are always ASCII (42, 3.14, 'hello', [], {}). The generator never invents non-ASCII test data.

  • The return value of generate

    A Perl character string. It is ASCII unless your schema put non-ASCII text into it as described above.

LIMITATIONS

Representative values are heuristic

_representative_value picks a single value based on type and min/max constraints. It does not guarantee the chosen value exercises any particular code path, and it does not use the schema's enum or matches keys.

No round-trip with extract-schemas

The generator reads any conforming ATG YAML schema, but it does not verify that the schema accurately describes the actual function's signature. If the schema is stale, the generated benchmark may pass wrong argument types.

SEE ALSO

Benchmark
bin/benchmark-generator
App::Test::Generator

AUTHOR

Nigel Horne, <njh at nigelhorne.com>

FORMAL SPECIFICATION

This section describes each method in Z notation. It is for readers who want an exact, mathematical statement. You do not need it to use the module.

How to read it: a name ending in ? is an input, a name ending in ! is an output, and a name ending in ' is the value after the call. The symbol Ξ before the state name means "this call does not change the object".

The state of an object is one value, the schema given to new:

┌─ BenchmarkGenerator ──────────────────────────────
│ schema : HASHREF
├───────────────────────────────────────────────────
│ schema ≠ undef
└───────────────────────────────────────────────────

new

┌─ New ─────────────────────────────────────────────
│ BenchmarkGenerator′
│ schema? : VALUE
│ result! : BenchmarkGenerator
├───────────────────────────────────────────────────
│ pre   schema? ≠ undef ∧ ref(schema?) = 'HASH'
│ post  schema′ = schema?
│       result! = θ BenchmarkGenerator′
└───────────────────────────────────────────────────

schema? = undef          ⇒ croak 'schema is required'
ref(schema?) ≠ 'HASH'    ⇒ croak 'schema must be a hashref'

generate

NAME == { s : STRING | s matches /^[A-Za-z_]\w*(::[A-Za-z_]\w*)*$/ }

┌─ Generate ────────────────────────────────────────
│ Ξ BenchmarkGenerator
│ result! : STRING
│ variants : ℙ STRING
├───────────────────────────────────────────────────
│ pre   schema.module ≠ undef ∧ schema.function ≠ undef
│       schema.module = 'builtin' ∨ schema.module ∈ NAME
│       schema.function ∈ NAME
│ post  schema.transforms = ∅ ⇒ variants = { 'default' }
│       schema.transforms ≠ ∅ ⇒ variants = dom schema.transforms
│       #variants = max(1, #(dom schema.transforms))
│       result! contains exactly one cmpthese(...) call
│       ∀ v : variants • result! contains one timed call named v
│       schema.module = 'builtin' ⇔ result! has no 'use module' line
│       ('new' ∈ dom schema ∧ schema.module ≠ 'builtin')
│           ⇔ result! builds $obj with module->new(...)
└───────────────────────────────────────────────────

STATE DIAGRAM

An object of this class has only one working state. It is created ready to use and it never changes afterwards.

         new(schema => undef)
         new(schema => not a hashref)
       +------------------------------+
       |  action: croak, no object    |
       v                              |
+-------------+                       |
|  NO OBJECT  |-----------------------+
+-------------+
       |
       | new(schema => HASHREF)
       | action: keep a reference to the schema
       v
+-------------+   generate()  [schema is good]
|             |---------------------------------+
|    READY    |   action: return the script     |
|             |<--------------------------------+
|             |
|             |   generate()  [module or function
|             |                missing or not a name]
|             |---------------------------------+
|             |   action: croak, nothing changes|
|             |<--------------------------------+
+-------------+
       |
       | last reference goes away
       | action: Perl frees the object
       v
+-------------+
|    GONE     |
+-------------+

Side effects: new and generate write no files and print nothing. The schema that you passed to new is never changed.

LICENCE AND COPYRIGHT

Copyright 2026 Nigel Horne.

Usage is subject to the terms of GPL2. If you use it, please let me know.