NAME
App::Test::Generator::BenchmarkGenerator - Generate Benchmark harnesses from ATG schemas
VERSION
Version 0.49
SYNOPSIS
Pattern 1: read a schema file and print the benchmark script.
use App::Test::Generator::BenchmarkGenerator;
use YAML::XS qw(LoadFile);
my $schema = LoadFile('schemas/my_func.yml');
my $bg = App::Test::Generator::BenchmarkGenerator->new(schema => $schema);
print $bg->generate();
Pattern 2: write the schema by hand, for a plain function that takes its arguments in a fixed order, and save the script to a file.
my $bg = App::Test::Generator::BenchmarkGenerator->new(
schema => {
module => 'List::Util',
function => 'max',
input => {
first => { type => 'integer', position => 0, min => 1, max => 9 },
second => { type => 'integer', position => 1 },
},
},
);
open my $fh, '>', 'benchmarks/max.pl' or die $!;
print $fh $bg->generate(); # the call is: List::Util::max(5, 42)
close $fh;
Pattern 3: compare several kinds of input for an object method. Each entry under transforms becomes one line in the timing table.
my $bg = App::Test::Generator::BenchmarkGenerator->new(
schema => {
module => 'My::Shop',
function => 'price',
new => { currency => 'GBP' },
input => { amount => { type => 'number' } },
transforms => {
small => { input => { amount => { type => 'number', max => 10 } } },
large => { input => { amount => { type => 'number', min => 1000 } } },
},
},
);
my $script = $bg->generate();
# The script builds: my $obj = My::Shop->new(currency => 'GBP');
# and then times: $obj->price(amount => 9)
# $obj->price(amount => 1001)
Pattern 4: benchmark a Perl built-in function. No use line is written for the special module name builtin.
my $bg = App::Test::Generator::BenchmarkGenerator->new(
schema => {
module => 'builtin',
function => 'abs',
input => { n => { type => 'number', position => 0 } },
},
);
print $bg->generate(); # the call is: abs(42)
DESCRIPTION
Given an ATG YAML schema (as produced by extract-schemas or written by hand), generates a self-contained Perl benchmark script using "cmpthese" in Benchmark.
Each transform defined in the schema becomes one variant in the cmpthese call, with representative input values derived from the transform's type and range constraints. When no transforms are defined, a single 'default' variant is emitted using the base input specification.
The generated script is a plain .pl file suitable for running directly with perl. It is not a test file and has no dependency on any test framework.
METHODS
new
Construct a new BenchmarkGenerator for a given ATG schema.
my $bg = App::Test::Generator::BenchmarkGenerator->new(schema => $schema);
Arguments
schemaA hashref representing the parsed YAML schema for the target function, as produced by
extract-schemasor written by hand. Must contain at minimummoduleandfunctionkeys. Required.
Returns
A blessed App::Test::Generator::BenchmarkGenerator object. Croaks if schema is missing or not a hashref.
EXAMPLE
use YAML::XS qw(LoadFile);
use App::Test::Generator::BenchmarkGenerator;
my $schema = LoadFile('schemas/greet.yml');
my $bg = App::Test::Generator::BenchmarkGenerator->new(schema => $schema);
print $bg->generate();
MESSAGES
schema is required-
schemawas not supplied or wasundef. schema must be a hashref-
schemawas supplied but is not a plain hashref (e.g. an arrayref or string was passed instead).
API SPECIFICATION
input
{
schema => { type => 'hashref' },
}
output
{
type => 'object',
isa => 'App::Test::Generator::BenchmarkGenerator',
}
The mathematical description is in "FORMAL SPECIFICATION".
generate
Generate the complete benchmark script as a string.
my $script = $bg->generate();
write_file('benchmarks/greet.pl', $script);
Arguments
None beyond $self.
Returns
A string containing the complete, self-contained Perl benchmark script using Benchmark::cmpthese. The string is ready to write directly to a .pl file and run with perl.
Croaks if the schema is missing a module or function key, or if either name is not a valid Perl name.
The object is not changed by this call. You can call generate many times and get the same text each time.
EXAMPLE
my $script = $bg->generate();
# Write to file
open my $fh, '>', 'benchmarks/my_func.pl' or die $!;
print $fh $script;
close $fh;
# Or run immediately:
require File::Temp;
my $tmp = File::Temp->new(SUFFIX => '.pl');
print $tmp $script;
system($^X, "$tmp");
MESSAGES
schema missing module-
The schema hashref has no
modulekey. schema missing function-
The schema hashref has no
functionkey. BenchmarkGenerator: module 'NAME' is not a valid Perl identifier-
The
modulevalue is not a Perl package name such asMy::Module. The special namebuiltinis always accepted. BenchmarkGenerator: function 'NAME' is not a valid Perl identifier-
The
functionvalue is not a Perl name such asgreetorMy::Module::greet. BenchmarkGenerator: cannot write a TYPE reference as a literal-
A value under
newis a code reference, an object or another kind of reference that cannot be written as Perl source. Plain values, arrayrefs and hashrefs are accepted.
API SPECIFICATION
input
None (reads from the schema passed to new).
output
{ type => 'string' }
The string is the complete benchmark script, ready to write to a file.
The mathematical description is in "FORMAL SPECIFICATION".
PSEUDOCODE
read module, function, input, transforms from schema
emit shebang, use strict/warnings, use Benchmark
unless module eq 'builtin': emit use $module
if schema has 'new' key and not builtin:
emit my $obj = Module->new(...)
if transforms defined and non-empty:
for each transform: build a named variant call
else:
build a single 'default' variant call
emit cmpthese($COUNT, { variant => sub { ... }, ... })
COMMON PITFALLS
- Schema missing
moduleorfunction -
generatecroaks immediately if either key is absent. Always ensure the schema has been loaded from a valid ATG YAML file before callinggenerate. - Expecting test-framework output
-
The generated script uses
Benchmark::cmptheseand prints timing results to STDOUT. It is not a test file and produces no TAP output. Do not run it withprove. - OOP schemas need a
new:key -
If the function under benchmark is an instance method, the schema must have a
new:key (even if its value is an empty hashref{}) sogenerateemits amy $obj = Module-new(...)> constructor call before thecmptheseblock. Without it, the variant calls useModule::function(...)form. - Transforms that omit the
inputkey -
If a transform's hashref has no
inputkey,generatefalls back to the base schemainputspec for that variant. This is intentional but may produce identical argument lists across variants if you forgot to add per- transform input overrides. - A transform
inputreplaces the baseinput. It is not merged. -
If the base
inputhas the parametersaandb, and a transforminputhas onlya, then that variant is called withaalone.bis not copied from the base. Repeat every parameter you need inside each transform. - Only plain data is allowed under
new -
A value under
newcan be a number, a string,undef, an arrayref or a hashref, nested to any depth. A code reference or an object makesgeneratecroak. - How
undefis handled -
schema => undefmakesnewcroak withschema is required.A
moduleorfunctionthat isundefmakesgeneratecroak.A missing or
undefinputortransformskey means "no parameters" or "no transforms". This is safe.new => undef(new: ~in YAML) still counts as "has a constructor". The script callsModule->new()with no arguments.An
undefvalue insidenewis written as the Perl wordundef.A parameter whose specification is
undefis passed as the Perl wordundef.
- The text
undefmeans "no input" -
Some ATG schema files contain
input: undef. YAML reads this as the five-letter string"undef".generatetreats that string, and any otherinputortransformsvalue that is not a hashref, as "no parameters". Inside a transform,input: undefgives a call with no arguments; it does not fall back to the baseinput. - Names that are not plain words are quoted for you
-
Transform names, parameter names and the keys under
newcan contain any characters. They are quoted and escaped in the generated script when needed. Onlymoduleandfunctionmust be valid Perl names. - Which text is written as a number
-
A string under
newis written without quotes only when it is a plain decimal number such as42,-1.5or2e10. Everything else, including'Inf','NaN'and' 12', is written as a quoted string. Be careful with a leading zero:'007'is written as007, which Perl reads as an octal number.
ENCODING
moduleandfunctionUse ASCII letters, digits, underscores and
::only. Non-ASCII letters may pass the name check, but the generated script has nouse utf8line, so Perl will not read such names correctly.Transform names, parameter names and string values under
newFull UTF-8 text, including non-ASCII letters and emojis, is copied into the script unchanged. This is safe only if you take two steps yourself: write the script through a UTF-8 file handle (
open my $fh, '>:encoding(UTF-8)', $path), and remember that the script has nouse utf8line, so the running benchmark sees the text as bytes, not as characters. For a benchmark that needs real character strings, adduse utf8;to the generated file by hand.Sample argument values
These are always ASCII (
42,3.14,'hello',[],{}). The generator never invents non-ASCII test data.The return value of
generateA Perl character string. It is ASCII unless your schema put non-ASCII text into it as described above.
LIMITATIONS
- Representative values are heuristic
-
_representative_valuepicks a single value based on type andmin/maxconstraints. It does not guarantee the chosen value exercises any particular code path, and it does not use the schema'senumormatcheskeys. - No round-trip with
extract-schemas -
The generator reads any conforming ATG YAML schema, but it does not verify that the schema accurately describes the actual function's signature. If the schema is stale, the generated benchmark may pass wrong argument types.
SEE ALSO
- Benchmark
bin/benchmark-generator- App::Test::Generator
AUTHOR
Nigel Horne, <njh at nigelhorne.com>
FORMAL SPECIFICATION
This section describes each method in Z notation. It is for readers who want an exact, mathematical statement. You do not need it to use the module.
How to read it: a name ending in ? is an input, a name ending in ! is an output, and a name ending in ' is the value after the call. The symbol Ξ before the state name means "this call does not change the object".
The state of an object is one value, the schema given to new:
┌─ BenchmarkGenerator ──────────────────────────────
│ schema : HASHREF
├───────────────────────────────────────────────────
│ schema ≠ undef
└───────────────────────────────────────────────────
new
┌─ New ─────────────────────────────────────────────
│ BenchmarkGenerator′
│ schema? : VALUE
│ result! : BenchmarkGenerator
├───────────────────────────────────────────────────
│ pre schema? ≠ undef ∧ ref(schema?) = 'HASH'
│ post schema′ = schema?
│ result! = θ BenchmarkGenerator′
└───────────────────────────────────────────────────
schema? = undef ⇒ croak 'schema is required'
ref(schema?) ≠ 'HASH' ⇒ croak 'schema must be a hashref'
generate
NAME == { s : STRING | s matches /^[A-Za-z_]\w*(::[A-Za-z_]\w*)*$/ }
┌─ Generate ────────────────────────────────────────
│ Ξ BenchmarkGenerator
│ result! : STRING
│ variants : ℙ STRING
├───────────────────────────────────────────────────
│ pre schema.module ≠ undef ∧ schema.function ≠ undef
│ schema.module = 'builtin' ∨ schema.module ∈ NAME
│ schema.function ∈ NAME
│ post schema.transforms = ∅ ⇒ variants = { 'default' }
│ schema.transforms ≠ ∅ ⇒ variants = dom schema.transforms
│ #variants = max(1, #(dom schema.transforms))
│ result! contains exactly one cmpthese(...) call
│ ∀ v : variants • result! contains one timed call named v
│ schema.module = 'builtin' ⇔ result! has no 'use module' line
│ ('new' ∈ dom schema ∧ schema.module ≠ 'builtin')
│ ⇔ result! builds $obj with module->new(...)
└───────────────────────────────────────────────────
STATE DIAGRAM
An object of this class has only one working state. It is created ready to use and it never changes afterwards.
new(schema => undef)
new(schema => not a hashref)
+------------------------------+
| action: croak, no object |
v |
+-------------+ |
| NO OBJECT |-----------------------+
+-------------+
|
| new(schema => HASHREF)
| action: keep a reference to the schema
v
+-------------+ generate() [schema is good]
| |---------------------------------+
| READY | action: return the script |
| |<--------------------------------+
| |
| | generate() [module or function
| | missing or not a name]
| |---------------------------------+
| | action: croak, nothing changes|
| |<--------------------------------+
+-------------+
|
| last reference goes away
| action: Perl frees the object
v
+-------------+
| GONE |
+-------------+
Side effects: new and generate write no files and print nothing. The schema that you passed to new is never changed.
LICENCE AND COPYRIGHT
Copyright 2026 Nigel Horne.
Usage is subject to the terms of GPL2. If you use it, please let me know.