NAME
App::Test::Generator - Fuzz Testing, Mutation Testing, LCSAJ Metrics and Test Dashboard for Perl modules
VERSION
Version 0.49
SYNOPSIS
App::Test::Generator is a suite to help the testing of CPAN modules. It consists of 6 subsystems:
Fuzz Tester
Mutation Testing
LCSAJ Metrics
Test Dashboard
Benchmark Generation
Workflow Deployment
From the command line:
# Takes the formal definition of a routine, creates tests against that routine, and runs the test
fuzz-harness-generator -r t/conf/abs.yml
# Attempt to create a formal definition from a routine package, then run tests against that formal definition
# This is the Holy Grail of test generation, a set of tests is automatically created directly from the source code,
extract-schemas lib/App/Test/Generator/Sample/Module.pm && fuzz-harness-generator -r schemas/greet.yml
# Fuzz a module and keep the corpus bounded: trim to the minimum subset that still covers every branch
extract-schemas --fuzz --minimize-corpus lib/My/Module.pm
# Generate round-trip tests that run every code example in a module's POD and verify the results
pod-example-tester lib/My/Module.pm --output t/pod_examples.t
# Generate a Benchmark::cmpthese script from a schema; each transform becomes one timed variant
benchmark-generator -i schemas/abs.yml -o benchmarks/abs.pl
# Copy dashboard.yml and mutate.yml into a module repository's .github/workflows/ directory
deploy-workflows --target /path/to/my-module
From Perl:
use App::Test::Generator qw(generate);
use App::Test::Generator::SchemaExtractor;
# Generate to STDOUT
App::Test::Generator->generate("t/conf/abs.yml");
# Generate directly to a file
App::Test::Generator->generate('t/conf/abs.yml', 't/add_fuzz.t');
# Holy grail mode - read a Perl file, generate tests, and run them
# This is a long way away yet, but see t/schema_input.t for a proof of concept
my $extractor = App::Test::Generator::SchemaExtractor->new(
input_file => 'lib/App/Test/Generator/Template.pm',
output_dir => '/tmp',
);
my $schemas = $extractor->extract_all();
use File::Temp qw(tempfile);
foreach my $schema(keys %{$schemas}) {
my ($fh, $tempfile) = tempfile(SUFFIX => '.t', UNLINK => 1);
close $fh;
App::Test::Generator->generate(
schema => $schemas->{$schema},
output_file => $tempfile,
);
system($^X, '-Ilib', $tempfile);
}
OVERVIEW
This module takes a formal input/output specification for a routine or method and automatically generates test cases. In effect, it allows you to easily add comprehensive black-box tests in addition to the more common white-box tests that are typically written for CPAN modules and other subroutines.
The generated tests combine:
Random fuzzing based on input types
Deterministic edge cases for min/max constraints
Static corpus tests defined in Perl or YAML
This approach strengthens your test suite by probing both expected and unexpected inputs, helping you to catch boundary errors, invalid data handling, and regressions without manually writing every case.
TOOLS
The distribution ships the following command-line tools:
benchmark-generator - generate a self-contained Benchmark
cmpthesescript from a YAML schema. Each transform in the schema becomes one named variant; representative input values are derived from each parameter's type and range constraints.deploy-workflows - copy
dashboard.ymlandmutate.ymlinto the target repository's.github/workflows/directory. Both files are embedded verbatim in the script, so no ATG source tree is needed after installation. Supports--target,--force, and--dry-run.To add a test dashboard to your CPAN module: copy these scripts into your
.github/workflowsdirectory, then commit the changes to GitHub and enable the page throughSettings-Pages-branch = gh_pages:extract-schemas - heuristically extract YAML parameter schemas from a
.pmfile, with optional coverage-guided fuzzing (--fuzz) and corpus minimization (--minimize-corpus).fuzz-harness-generator - generate a
Test::Mostfuzzing harness from a YAML schema.pod-example-tester - generate a
Test::Mostround-trip test file from a module's POD code examples. Annotated examples (# returns value/# => value) getis()assertions; unannotated verbatim blocks are wrapped ineval{}and checked for no exception.test-generator-mutate - run mutation testing against a module's test suite.
test-generator-index - generate the HTML test-quality dashboard, combining Devel::Cover statement/branch data, LCSAJ path coverage, mutation results, and CPAN Testers failure analysis. For each CPAN Testers FAIL report, also writes a self-contained shell script (
cover_html/reproduce/reproduce-GUID.sh) that pins every installed module at its exact failing version, enabling local reproduction of the failure environment.
DESCRIPTION
This module implements the logic behind fuzz-harness-generator. It parses configuration files (fuzz and/or corpus YAML), and produces a ready-to-run .t test script to run through prove.
It reads configuration files in any format, and optional YAML corpus files. All of the examples in this documentation are in YAML format, other formats may not work as they aren't so heavily tested. It then generates a Test::Most-based fuzzing harness combining:
Randomized fuzzing of inputs (with edge cases)
Optional static corpus tests from Perl
%casesor YAML file (yaml_caseskey)Functional or OO mode (via
$new)Reproducible runs via
$seedand configurable iterations via$iterations
MUTATION-GUIDED TEST GENERATION
App::Test::Generator includes a pipeline that automatically closes the feedback loop between mutation testing, schema extraction, and fuzz testing. The goal is that surviving mutants drive the creation of new tests that kill them on the next run, without manual intervention.
The Pipeline
mutation survivor
|
v
SchemaExtractor extracts the schema for the enclosing sub
|
v
Schema augmented with boundary values from the mutant
|
v
Augmented schema written to t/conf/
|
v
t/fuzz.t picks up the new schema and runs fuzz tests
|
v
Mutation killed on next run
How to Use It
The pipeline is driven by three flags passed to bin/test-generator-index, which is invoked automatically by bin/generate-test-dashboard on each CI push.
Step 1: Generate TODO stubs for all survivors
bin/test-generator-index --generate_mutant_tests=t
Produces t/mutant_YYYYMMDD_HHMMSS.t containing:
TODO stubs for HIGH and MEDIUM difficulty survivors, with boundary value suggestions, environment variable hints, and the enclosing subroutine name for navigation context.
Comment-only hints for LOW difficulty survivors.
Multiple mutations on the same source line are deduplicated into one stub. One good test kills all variants on that line.
Step 2: Generate runnable schemas for NUM_BOUNDARY survivors
bin/test-generator-index \
--generate_mutant_tests=t \
--generate_test=mutant
For each NUM_BOUNDARY survivor, calls App::Test::Generator::SchemaExtractor to extract the schema for the enclosing subroutine. If the confidence level is sufficient, the schema is augmented with the boundary value from the mutant (plus one value either side) and written to t/conf/ as a runnable YAML file. "fuzz.t" in t picks it up automatically on the next test run.
Falls back to a TODO stub if:
SchemaExtractor cannot parse the file
The enclosing sub cannot be determined
The extracted schema confidence is
very_lowornone
Step 3: Augment existing schemas with survivor boundary values
bin/test-generator-index \
--generate_mutant_tests=t \
--generate_test=mutant \
--generate_fuzz
Scans t/conf/ for existing YAML schema files (hand-written or previously generated) and writes augmented copies with boundary values from surviving NUM_BOUNDARY mutants merged in. The original schema is never modified. Augmented copies are written as t/conf/mutant_fuzz_YYYYMMDD_HHMMSS_FUNCTION.yml and picked up automatically by t/fuzz.t.
Schemas whose filename already starts with mutant_fuzz_ are skipped to prevent cascading augmentation. Schemas with no matching survivors are skipped, with a note if --verbose is active.
Putting It All Together
The recommended invocation in bin/generate-test-dashboard Step 7 runs all three stages together:
bin/test-generator-index \
--generate_mutant_tests=t \
--generate_test=mutant \
--generate_fuzz
The GitHub Actions workflow in .github/workflows/dashboard.yml then commits any new t/mutant_*.t and t/conf/mutant_*.yml files to the repository so they accumulate over time as the test suite improves.
Confidence Levels
App::Test::Generator::SchemaExtractor assigns a confidence level to each extracted schema:
high/medium/low- Schema is used for test generationvery_low/none- Falls back to TODO stub
Confidence is based on how much type and constraint information could be inferred from the source code and its POD documentation. Methods with explicit parameter validation (Params::Validate::Strict, Params::Get) or comprehensive POD will produce higher-confidence schemas.
Files Produced
t/mutant_YYYYMMDD_HHMMSS.tTODO stub file for all survivors. Committed to the repository by the GitHub Actions workflow.
t/conf/mutant_MODNAME_FUNCTION_YYYYMMDD_HHMMSS.ymlRunnable YAML schema for a NUM_BOUNDARY survivor where SchemaExtractor confidence was sufficient. Picked up by
t/fuzz.t.t/conf/mutant_fuzz_YYYYMMDD_HHMMSS_FUNCTION.ymlAugmented copy of an existing schema with survivor boundary values merged in. Picked up by
t/fuzz.t.
See Also
App::Test::Generator::SchemaExtractor - Schema extraction from Perl source code
"test-generator-index" in bin - Dashboard generator and pipeline driver
"generate-test-dashboard" in bin - Full pipeline runner
CONFIGURATION
The configuration file, for each set of tests to be produced, is a file containing a schema that can be read by Config::Abstraction.
SCHEMA
The schema is split into several sections.
%input - input params with keys => type/optional specs
When using named parameters
input:
name:
type: string
optional: false
age:
type: integer
optional: true
Supported basic types used by the fuzzer: string, integer, float, number, boolean, arrayref, hashref. See also Params::Validate::Strict. You can add more custom types using properties.
A nested schema describes the elements of an arrayref or the values of a hashref. Element schemas are full specs, so they carry their own constraints and can nest again:
input:
words:
type: arrayref
min: 1
max: 3
schema:
type: string
min: 3
max: 10
scores:
type: hashref
schema:
type: integer
min: 0
max: 100
record:
type: hashref
schema:
name:
type: string
min: 2
max: 20
age: integer
grid:
type: arrayref
schema:
type: arrayref
min: 1
max: 2
schema:
type: integer
min: 0
max: 9
things:
type: arrayref
schema:
type: object
isa: My::Thing # needs a no-argument constructor
# or: can: label # mocked with Class::Simple
A hashref schema without a type is a map of fixed key names to element schemas, so generated valid data uses those keys instead of random filler. Generated element values satisfy the element's own constraints.
For routines with one unnamed parameter
input:
type: string
For routines with more than one named parameter, use the position keyword.
module: Math::Simple::MinMax
fuction: max
input:
left:
type: number
position: 0
right:
type: number
position: 1
output:
type: number
The keyword undef is used to indicate that the function takes no arguments.
%output - output param types for Return::Set checking
output:
type: string
If the output hash contains the key _STATUS, and if that key is set to DIES, the routine should die with the given arguments; otherwise, it should live. If it's set to WARNS, the routine should warn with the given arguments. The output can be set to the string 'undef' if the routine should return the undefined value:
---
module: Scalar::Util
function: blessed
input:
type: string
output: undef
The keyword undef is used to indicate that the function returns nothing.
For methods that return a list (rather than a reference), use type: array. The generated test captures the result in list context and validates it as an arrayref, which requires Test::Returns 0.03 or later:
output:
type: array
%config - optional hash of configuration.
The current supported variables are
close_stdinTests should not attempt to read from STDIN (default: 1). This is ignored on Windows, when never closes STDIN.
test_nuls, inject NUL bytes into strings (default: 1)With this test enabled, the function is expected to die when a NUL byte is passed in.
test_undef, test with undefined value (default: 1)test_empty, test with empty strings (default: 1)test_non_ascii, test with strings that contain non ascii characters (default: 1)timeout, ensure tests don't hang (default: 10)regex_timeout, seconds allowed for generating one string that matches a regex pattern (default: 5)A pattern the generator cannot satisfy, or one that makes it search for too long, falls back to a fixed matching value; when none exists the pattern's cases are skipped and reported once instead of failing the run. Matching strings are generated from the seeded RNG, so a schema's
seedmakes them reproducible.Setting this to 0 disables timeout testing.
check_files, assert the routine leaves no files in the working directory (default: 1)Every call is checked for files it created. They are removed afterwards whatever the setting, so a routine that writes a file cannot break the cases that follow it; set this to 0 for a routine whose purpose is to write one.
extract-schemassets it to 0 when the routine's POD documents creating or writing a file in its SIDE EFFECTS section.check_fds, assert the routine leaks no file descriptors (default: 1)Only checked where /proc/self/fd exists. Set this to 0 for a routine that returns a filehandle to its caller, which still holds a descriptor.
The first descriptor a call opens on a given file (or its first socket or pipe) is taken as a resource held on purpose, such as a log file an object opens on first use, and is not reported. A later call that opens the same target again is.
check_accessors, assert an accessor leaves the object it was called on alone (default: 1)The object is deep copied before each call and the state it was handed compared afterwards. A getter that overwrites or deletes that state fails "getter leaves its object unmodified"; a getset that changes anything but the property it is named for fails "setter changes nothing but its property". Values the call fills in where it was handed undef, and keys it adds, are ignored, so a getter that memoises what it computed is not reported. Set this to 0 for an accessor whose purpose is to change state, such as an iterator's
next(); a getter that declares no input is called with no arguments either way.dedup, fuzzing can create duplicate tests, go some way to remove duplicates (default: 1)properties, enable Test::LectroTest Property tests (default: 0)*item *
test_security, send some security string based tests (default: 0)
All values default to true.
%accessor - this is an accessor routine
accessor:
property: ua
type: getset
Has two mandatory elements:
propertyThe name of the property in the object that the routine controls.
typeOne of
getter,setter,getset.
%transforms - list of transformations from input sets to output sets
Transforms allow you to define how input data should be transformed into output data. This is useful for testing functions that convert between formats, normalize data, or apply business logic transformations on a set of data to different set of data. It takes a list of subsets of the input and output definitions, and verifies that data from each input subset is correctly transformed into data from the matching output subset.
A transform whose input names at least one field that the top-level input does not introduces its own call shape: its test cases are built from those fields rather than the top-level ones. This lets one schema cover two calls, such as an arrayref argument and a list of scalar arguments. A transform whose input only names fields that are already at the top level adjusts the top-level input and keeps the usual behaviour.
Transform Validation Rules
For each transform:
- 1. Generate test cases using the transform's input schema
- 2. Call the function with those inputs
- 3. Validate the output matches the transform's output schema
- 4. If output has a specific 'value', check exact match
- 5. If output has constraints (min/max), validate within bounds
Example 1
---
module: builtin
function: abs
config:
test_undef: no
test_empty: no
test_nuls: no
test_non_ascii: no
input:
number:
type: number
position: 0
output:
type: number
min: 0
transforms:
positive:
input:
number:
type: number
position: 0
min: 0
output:
type: number
min: 0
negative:
input:
number:
type: number
position: 0
max: 0
output:
type: number
min: 0
error:
input:
undef
output:
_STATUS: DIES
If the output hash contains the key _STATUS, and if that key is set to DIES, the routine should die with the given arguments; otherwise, it should live. If it's set to WARNS, the routine should warn with the given arguments.
The keyword undef is used to indicate that the function returns nothing.
Example 2
---
module: Math::Utils
function: normalize_number
input:
value:
type: number
position: 0
output:
type: number
transforms:
positive_stays_positive:
input:
value:
type: number
min: 0
max: 1000
output:
type: number
min: 0
max: 1
negative_becomes_zero:
input:
value:
type: number
max: 0
output:
type: number
value: 0
preserves_zero:
input:
value:
type: number
value: 0
output:
type: number
value: 0
$module
The name of the module (optional).
Using the reserved word builtin means you're testing a Perl builtin function.
If omitted, the generator will guess from the config filename: My-Widget.conf -> My::Widget.
$function
The function/method to test.
This defaults to run.
%new
An optional hashref of args to pass to the module's constructor.
new:
api_key: ABC123
verbose: true
To ensure new() is called with no arguments, you still need to define new, thus:
module: MyModule
function: my_function
new:
%cases
An optional Perl static corpus, when the output is a simple string (expected => [ args... ]).
Maps the expected output string to the input and _STATUS
cases:
ok:
input: ping
_STATUS: OK
error:
input: ""
_STATUS: DIES
$yaml_cases
An optional path to a YAML file with the same shape as %cases.
$seed
An optional integer. When provided, the generated t/fuzz.t will call srand($seed) so fuzz runs are reproducible.
$iterations
An optional integer controlling how many fuzz iterations to perform (default 30).
%edge_cases
An optional hash mapping of extra values to inject.
# Two named parameters
edge_cases:
name: [ '', 'a' x 1024, \"\x{263A}" ]
age: [ -1, 0, 99999999 ]
# Takes a string input
edge_cases: [ 'foo', 'bar' ]
Values can be strings or numbers; strings will be properly quoted. Note that this only works with routines that take named parameters.
%type_edge_cases
An optional hash mapping types to arrayrefs of extra values to try for any field of that type:
type_edge_cases:
string: [ '', ' ', "\t", "\n", "\0", 'long' x 1024, chr(0x1F600) ]
number: [ 0, 1.0, -1.0, 1e308, -1e308, 1e-308, -1e-308, 'NaN', 'Infinity' ]
integer: [ 0, 1, -1, 2**31-1, -(2**31), 2**63-1, -(2**63) ]
%edge_case_array
Specify edge case values for routines that accept a single unnamed parameter. This is specifically designed for simple functions that take one argument without a parameter name. These edge cases supplement the normal random string generation, ensuring specific problematic values are always tested. During fuzzing iterations, there's a 40% probability that a test case will use a value from edge_case_array instead of randomly generated data.
---
module: Text::Processor
function: sanitize
input:
type: string
min: 1
max: 1000
edge_case_array:
- "<script>alert('xss')</script>"
- "'; DROP TABLE users; --"
- "\0null\0byte"
- "emoji\U0001F60Atest"
- ""
- " "
seed: 42
iterations: 30
Semantic Data Generators
For property-based testing with Test::LectroTest, you can use semantic generators to create realistic test data.
unix_timestamp is currently fully supported, other fuzz testing support for semantic entries is being developed.
input:
email:
type: string
semantic: email
user_id:
type: string
semantic: uuid
phone:
type: string
semantic: phone_us
Available Semantic Types
email- Valid email addresses (user@domain.tld)url- HTTP/HTTPS URLsuuid- UUIDv4 identifiersphone_us- US phone numbers (XXX-XXX-XXXX)phone_e164- International E.164 format (+XXXXXXXXXXXX)ipv4- IPv4 addresses (0.0.0.0 - 255.255.255.255)ipv6- IPv6 addressesusername- Alphanumeric usernames with _ and -slug- URL slugs (lowercase-with-hyphens)hex_color- Hex color codes (#RRGGBB)iso_date- ISO 8601 dates (YYYY-MM-DD)iso_datetime- ISO 8601 datetimes (YYYY-MM-DDTHH:MM:SSZ)semver- Semantic version strings (major.minor.patch)jwt- JWT-like tokens (base64url format)json- Simple JSON objectsbase64- Base64-encoded stringsmd5- MD5 hashes (32 hex chars)sha256- SHA-256 hashes (64 hex chars)directory- The path of an existing, writable directoryThe harness creates a temporary directory and passes its path, because no pattern can describe "a directory that exists". No empty-string case is generated for such a parameter, since routines commonly read an empty path as "not given". App::Test::Generator::SchemaExtractor sets this for a parameter that the routine tests with
-d.existing_file- The path of an existing fileThe harness creates a temporary file and passes its path, because no pattern can describe "a file that exists". App::Test::Generator::SchemaExtractor sets this for a parameter that the routine tests with
-eor-fbut does not read.unix_timestamp
EDGE CASE GENERATION
In addition to purely random fuzz cases, the harness generates deterministic edge cases for parameters that declare min, max or len in their schema definitions.
For each constraint, three edge cases are added:
Just inside the allowable range
This case should succeed, since it lies strictly within the bounds.
Exactly on the boundary
This case should succeed, since it meets the constraint exactly.
Just outside the boundary
This case is annotated with
_STATUS = 'DIES'in the corpus and should cause the harness to fail validation or croak.A constraint that came only from extract-schemas reading POD (marked with
_source: podfor prose or_source: input_specfor a formal spec) is a claim about valid input, not a check the routine has been seen to make. The constraint still sizes the valid data that is generated. It also gets explicit out-of-bound probes: a probe that dies is accepted as enforcement of the documented limit, while one that survives is recorded as a documentation contradiction. Provenance is tracked per constraint in_constraint_source, so a documented minimum does not make a maximum that the code enforces tolerant on the same field. This applies to array and hash sizes, string lengths and numeric ranges, to constraints parsed from POD prose (3-50,at least 5), and tomemberoflists andmatchespatterns. Every documented constraint gets an outside value as an advisory probe; in the normal advisory mode a probe that survives is reported as a routine accepting a value its POD says is invalid, without failing the run. A constraint written by hand in a schema is still required to die outside it. Because tolerance reduces what the harness proves, every tolerated bound is reported withdiag, so a passing run never hides the concession. Without this, a routine whose POD saysmin => 1but which accepts an empty list failed every harness run.Fuzzing can still reveal a contradiction in either direction: a routine may reject a value inside a documented constraint, or accept one outside it. The harness records those observations and reports them together at the end of the run. Advisory mode tolerates them, so POD-derived documentation cannot fail a routine that merely documents differently; pass
--strict-documentedtofuzz-harness-generatorto make each recorded contradiction a failing test. The strict choice is embedded in the generated harness, so a laterproverun behaves the same way.
Supported constraint types:
number,integer,floatUses numeric values one below, equal to, and one above the boundary.
stringUses strings of lengths one below, equal to, and one above the boundary.
arrayrefUses references to arrays of with the number of elements one below, equal to, and one above the boundary.
hashrefUses hashes with key counts one below, equal to, and one above the boundary (
min= minimum number of keys,max= maximum number of keys).memberof- arrayref of allowed values for a parameterThis example is for a routine called
input()that takes two arguments:statusandlevel.statusis a string that must have the valueok,errororpending. Thelevelargument is an integer that must be one of1,5or111.--- input: status: type: string memberof: - ok - error - pending level: type: integer memberof: - 1 - 5 - 111The generator will automatically create test cases for each allowed value (inside the member list), and at least one value outside the list (which should die or
croak,_STATUS = 'DIES'). This works for strings, integers, and numbers.enum- synonym ofmemberofboolean- automatic boundary tests for boolean fieldsinput: flag: type: booleanThe generator will automatically create test cases for 0 and 1; true and false; off and on, and values that should trigger
_STATUS = 'DIES'.
These edge cases are inserted automatically, in addition to the random fuzzing inputs, so each run will reliably probe boundary conditions without relying solely on randomness.
EXAMPLES
See the files in t/conf for examples.
Adding Scheduled fuzz Testing with GitHub Actions to Your Code
To automatically create and run tests on a regular basis on GitHub Actions, you need to create a configuration file for each method and subroutine that you're testing, and a GitHub Actions configuration file.
This example takes you through testing the online_render method of HTML::Genealogy::Map.
t/conf/online_render.yml
---
module: HTML::Genealogy::Map
function: onload_render
input:
gedcom:
type: object
can: individuals
geocoder:
type: object
can: geocode
debug:
type: boolean
optional: true
google_key:
type: string
optional: true
min: 39
max: 39
matches: "^AIza[0-9A-Za-z_-]{35}$"
config:
test_undef: 0
.github/actions/fuzz.t
---
name: Fuzz Testing
permissions:
contents: read
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]
schedule:
- cron: '29 5 14 * *'
jobs:
generate-fuzz-tests:
strategy:
fail-fast: false
matrix:
os:
- macos-latest
- ubuntu-latest
- windows-latest
perl: ['5.42', '5.40', '5.38', '5.36', '5.34', '5.32', '5.30', '5.28', '5.22']
runs-on: ${{ matrix.os }}
name: Fuzz testing with perl ${{ matrix.perl }} on ${{ matrix.os }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Set up Perl
uses: shogo82148/actions-setup-perl@8b574cdc2dffdae49f803204a4f2b716a2fa1db7 # v1.44.2
with:
perl-version: ${{ matrix.perl }}
- name: Install App::Test::Generator this module's dependencies
run: |
cpanm App::Test::Generator
cpanm --installdeps .
- name: Make Module
run: |
perl Makefile.PL
make
env:
AUTOMATED_TESTING: 1
NONINTERACTIVE_TESTING: 1
- name: Generate fuzz tests
run: |
mkdir t/fuzz
find t/conf -name '*.yml' | while read config; do
test_name=$(basename "$config" .conf)
fuzz-harness-generator "$config" > "t/fuzz/${test_name}_fuzz.t"
done
- name: Run generated fuzz tests
run: |
prove -lr t/fuzz/
env:
AUTOMATED_TESTING: 1
NONINTERACTIVE_TESTING: 1
Fuzz Testing your CPAN Module
Running fuzz tests when you run make test in your CPAN module.
Create a directory <t/conf> which contains the schemas.
Then create this file as <t/fuzz.t>:
#!/usr/bin/env perl
use strict;
use warnings;
use FindBin qw($Bin);
use IPC::Run3;
use IPC::System::Simple qw(system);
use Test::Needs 'App::Test::Generator';
use Test::Most;
my $dirname = "$Bin/conf";
if((-d $dirname) && opendir(my $dh, $dirname)) {
while (my $filename = readdir($dh)) {
# Skip '.' and '..' entries and vi temporary files
next if ($filename eq '.' || $filename eq '..') || ($filename =~ /\.swp$/);
my $filepath = "$dirname/$filename";
if(-f $filepath) { # Check if it's a regular file
my ($stdout, $stderr);
run3 ['fuzz-harness-generator', '-r', $filepath], undef, \$stdout, \$stderr;
ok($? == 0, 'Generated test script exits successfully');
if($? == 0) {
ok($stdout =~ /^Result: PASS/ms);
if($stdout =~ /Files=1, Tests=(\d+)/ms) {
diag("$1 tests run");
}
} else {
diag("$filepath: STDOUT:\n$stdout");
diag($stderr) if(length($stderr));
diag("$filepath Failed");
last;
}
diag($stderr) if(length($stderr));
}
}
closedir($dh);
}
done_testing();
Property-Based Testing with Transforms
The generator can create property-based tests using Test::LectroTest when the properties configuration option is enabled. This provides more comprehensive testing by automatically generating thousands of test cases and verifying that mathematical properties hold across all inputs.
Basic Property-Based Transform Example
Here's a complete example testing the abs builtin function:
t/conf/abs.yml:
---
module: builtin
function: abs
config:
test_undef: no
test_empty: no
test_nuls: no
properties:
enable: true
trials: 1000
input:
number:
type: number
position: 0
output:
type: number
min: 0
transforms:
positive:
input:
number:
type: number
min: 0
output:
type: number
min: 0
negative:
input:
number:
type: number
max: 0
output:
type: number
min: 0
This configuration:
Enables property-based testing with 1000 trials per property
Defines two transforms: one for positive numbers, one for negative
Automatically generates properties that verify
abs()always returns non-negative numbers
Generate the test:
fuzz-harness-generator t/conf/abs.yml > t/abs_property.t
The generated test will include:
Traditional edge-case tests for boundary conditions
Random fuzzing with 30 iterations (or as configured)
Property-based tests that verify the transforms with 1000 trials each
What Properties Are Tested?
The generator automatically detects and tests these properties based on your transform specifications:
Range constraints - If output has
minormax, verifies results stay within boundsType preservation - Ensures numeric inputs produce numeric outputs
Definedness - Verifies the function doesn't return
undefunexpectedlySpecific values - If output specifies a
value, checks exact equality
For the abs example above, the generated properties verify:
# For the "positive" transform:
- Given a positive number, abs() returns >= 0
- The result is a valid number
- The result is defined
# For the "negative" transform:
- Given a negative number, abs() returns >= 0
- The result is a valid number
- The result is defined
Advanced Example: String Normalization
Here's a more complex example testing a string normalization function:
t/conf/normalize.yml:
---
module: Text::Processor
function: normalize_whitespace
config:
properties:
enable: true
trials: 500
input:
text:
type: string
min: 0
max: 1000
position: 0
output:
type: string
min: 0
max: 1000
transforms:
empty_preserved:
input:
text:
type: string
value: ""
output:
type: string
value: ""
single_space:
input:
text:
type: string
min: 1
matches: '^\S+(\s+\S+)*$'
output:
type: string
matches: '^\S+( \S+)*$'
length_bounded:
input:
text:
type: string
min: 1
max: 100
output:
type: string
min: 1
max: 100
This tests that the normalization function:
Preserves empty strings (
empty_preservedtransform)Collapses multiple spaces into single spaces (
single_spacetransform)Maintains length constraints (
length_boundedtransform)
Interpreting Property Test Results
When property-based tests run, you'll see output like:
ok 123 - negative property holds (1000 trials)
ok 124 - positive property holds (1000 trials)
If a property fails, Test::LectroTest will attempt to find the minimal failing case and display it:
not ok 123 - positive property holds (47 trials)
# Property failed
# Reason: counterexample found
This helps you quickly identify edge cases that your function doesn't handle correctly.
Configuration Options for Property-Based Testing
In the config section:
config:
properties:
enable: true # Enable property-based testing (default: false)
trials: 1000 # Number of test cases per property (default: 1000)
You can also disable traditional fuzzing and only use property-based tests:
config:
properties:
enable: true
trials: 5000
iterations: 0 # Disable random fuzzing, use only property tests
When to Use Property-Based Testing
Property-based testing with transforms is particularly useful for:
Mathematical functions (
abs,sqrt,min,max, etc.)Data transformations (encoding, normalization, sanitization)
Parsers and formatters
Functions with clear input-output relationships
Code that should satisfy mathematical properties (commutativity, associativity, idempotence)
Requirements
Property-based testing requires both Test::LectroTest and Test::LectroTest::Compat to be installed:
cpanm Test::LectroTest Test::LectroTest::Compat
Test::LectroTest::Compat provides the use_ok bridge between Test::LectroTest and Test::Most; it is used in every generated property-based test file. Both are declared in the distribution's TEST_REQUIRES so they are installed automatically during make test.
If not installed, the generated tests will automatically skip the property-based portion with a message.
Testing Email Validation
---
module: Email::Valid
function: rfc822
config:
properties:
enable: true
trials: 200
close_stdin: true
test_undef: no
test_empty: no
test_nuls: no
input:
email:
type: string
semantic: email
position: 0
output:
type: boolean
transforms:
valid_emails:
input:
email:
type: string
semantic: email
output:
type: boolean
This generates 200 realistic email addresses for testing, rather than random strings.
Combining Semantic with Regex
You can combine semantic generators with regex validation:
input:
corporate_email:
type: string
semantic: email
matches: '@company\.com$'
The semantic generator creates realistic emails, and the regex ensures they match your domain.
Custom Properties for Transforms
You can define additional properties that should hold for your transforms beyond the automatically detected ones.
Using Built-in Properties
transforms:
positive:
input:
number:
type: number
min: 0
output:
type: number
min: 0
properties:
- idempotent # f(f(x)) == f(x)
- non_negative # result >= 0
- positive # result > 0
Available built-in properties:
idempotent- Function is idempotent: f(f(x)) == f(x)non_negative- Result is always >= 0positive- Result is always > 0non_empty- String result is never emptylength_preserved- Output length equals input lengthuppercase- Result is all uppercaselowercase- Result is all lowercasetrimmed- No leading/trailing whitespacesorted_ascending- Array is sorted ascendingsorted_descending- Array is sorted descendingunique_elements- Array has no duplicatespreserves_keys- Hash has same keys as input
Custom Property Code
Custom properties allows the definition additional invariants and relationships that should hold for their transforms, beyond what's auto-detected. For example:
Idempotence: f(f(x)) == f(x)
Commutativity: f(x, y) == f(y, x)
Associativity: f(f(x, y), z) == f(x, f(y, z))
Inverse relationships: decode(encode(x)) == x
Domain-specific invariants: Custom business logic
Define your own properties with custom Perl code:
transforms:
normalize:
input:
text:
type: string
output:
type: string
properties:
- name: single_spaces
description: "No multiple consecutive spaces"
code: $result !~ / /
- name: no_leading_space
description: "No space at start"
code: $result !~ /^\s/
- name: reversible
description: "Can be reversed back"
code: length($result) == length($text)
The code has access to:
$result- The function's return valueInput variables - All input parameters (e.g.,
$text,$number)The function itself - Can call it again for idempotence checks
Combining Auto-detected and Custom Properties
The generator automatically detects properties from your output spec, and adds your custom properties:
transforms:
sanitize:
input:
html:
type: string
output:
type: string
min: 0 # Auto-detects: defined, min_length >= 0
max: 10000
properties: # Additional custom checks:
- name: no_scripts
code: $result !~ /<script/i
- name: no_iframes
code: $result !~ /<iframe/i
GENERATED OUTPUT
The generated test:
Seeds RND (if configured) for reproducible fuzz runs
Uses edge cases (per-field and per-type) with configurable probability
Runs
$iterationsfuzz cases plus appended edge-case runsValidates inputs with Params::Get / Params::Validate::Strict
Asserts that a wrong-typed argument dies, where the type is enforced
Passing a reference where a string is declared is only expected to die for a parameter whose input spec sets
_source: validator. App::Test::Generator::SchemaExtractor sets that source when the routine checks its arguments with a validator such as Params::Validate::Strict. A type taken from POD, or inferred from the code, documents intent but does not promise rejection, so no such case is generated for it. Add the source by hand to a schema whose routine does check its argument types.Validates outputs with Return::Set
Runs static
is(... )corpus tests from Perl and/or YAML corpusRuns Test::LectroTest tests
METHODS
generate
Takes a schema file and produces a test file (or STDOUT).
# Modern named API
App::Test::Generator->generate(
schema_file => 'schemas/foo.yml',
output_file => 'test/foo.t',
);
# Feed surviving mutants into the generated harness
App::Test::Generator->generate(
schema_file => 'schemas/foo.yml',
mutation_results => 'mutation.json',
output_file => 'test/foo.t',
);
# Legacy positional API
App::Test::Generator->generate($schema_file, $test_file);
API SPECIFICATION
Input
{
schema_file => { type => 'string', optional => 0 },
input_file => { type => 'string', optional => 1 },
output_file => { type => 'string', optional => 1, max => 255 },
strict_documented => { type => 'boolean', optional => 1 },
mutation_results => { type => 'string', optional => 1 },
}
mutation_results is the JSON report written by test-generator-mutate --json. Surviving numeric-boundary mutants are converted into min or max constraints on the matching input field, so the harness generates data either side of the boundary the survivor exposed. Existing constraints are never replaced. See App::Test::Generator::Mutation::Feedback.
Output
{ type => 'boolean' }
Always 1. The test source itself is written to output_file, or printed on STDOUT when no output file is given; it is not returned.
The mathematical description is in "FORMAL SPECIFICATION".
render_fallback
Render any Perl value into a compact Perl source-code string using Data::Dumper. Used as a catch-all when no more specific renderer applies.
my $code = render_fallback({ key => 'value' });
# returns: "{'key' => 'value'}"
Arguments
$vAny Perl value, including undef, scalars, refs, and blessed objects.
Returns
A string of Perl source code that reproduces the value when evaluated. Returns the string 'undef' when $v is undef.
Side effects
Temporarily sets $Data::Dumper::Terse and $Data::Dumper::Indent to produce compact single-line output. Both are restored on return via local.
Notes
The output is always a single line with no trailing newline. Suitable for embedding in generated test code where readability is secondary to correctness.
API SPECIFICATION
input
{ v => { type => 'any', optional => 1 } }
output
{ type => 'string' }
The mathematical description is in "FORMAL SPECIFICATION".
render_hash
Render a two-level hashref (parameter name => spec hashref) into Perl source code suitable for embedding in a generated test file as the input specification passed to Params::Validate::Strict.
my $code = render_hash(\%input);
Arguments
$hrefA hashref whose values are themselves hashrefs containing field specifications. A scalar value that is a recognised type string (see
_valid_type) is expanded to{ type => $value }. Any other non-hashref value is skipped with a warning.
Returns
A string of comma-separated Perl source-code lines, one per key, of the form:
'key' => { subkey => value, ... }
Returns an empty string if $href is undef, empty, or not a hashref.
Notes
The matches and nomatch sub-keys are treated specially - their values are compiled to Regexp objects via eval { qr/.../ } and then rendered using perl_quote so they appear as qr{...} in the generated test. This prevents unmatched bracket characters in the pattern from causing compilation failures.
Other sub-keys are rendered via perl_quote.
API SPECIFICATION
input
{ href => { type => 'any', optional => 1 } }
output
{ type => 'string' }
The mathematical description is in "FORMAL SPECIFICATION".
render_args_hash
Render a flat hashref into a Perl source-code argument list of the form 'key' = value, ...>, suitable for embedding in a function call in a generated test file.
my $code = render_args_hash({ type => 'string', min => 1 });
# returns: "'min' => 1, 'type' => 'string'"
Arguments
$hrefA flat hashref of key-value pairs. Values may be scalars, arrayrefs, or Regexp objects - all are handled by
perl_quote.
Returns
A comma-separated string of key = value> pairs sorted by key. Returns an empty string if $href is undef, empty, or not a hashref.
Notes
Keys and values are both rendered via perl_quote. In particular, Regexp values are rendered as qr{...} which is correct for Params::Validate::Strict and Return::Set schema arguments in the generated test.
API SPECIFICATION
input
{ href => { type => 'any', optional => 1 } }
output
{ type => 'string' }
The mathematical description is in "FORMAL SPECIFICATION".
render_arrayref_map
Render a hashref whose values are arrayrefs into a Perl source-code fragment suitable for use as a hash literal in a generated test file.
my $code = render_arrayref_map({ name => ['', 'a' x 100] });
Arguments
$hrefA hashref whose values are arrayrefs. Keys whose values are not arrayrefs are silently skipped.
Returns
A comma-separated string of 'key' = [ val, ... ]> entries, one per qualifying key, sorted alphabetically. Returns the string '()' if $href is undef or not a hashref - this produces an empty hash assignment in the generated test rather than a syntax error. Returns an empty string if $href is a hashref with no arrayref values.
Notes
Array element values are rendered via perl_quote which handles scalars, arrayrefs, and Regexp objects. Non-arrayref values are skipped without warning - this is intentional since callers may pass mixed-value hashes and only want the arrayref entries rendered.
API SPECIFICATION
input
{ href => { type => 'any', optional => 1 } }
output
{ type => 'string' }
The mathematical description is in "FORMAL SPECIFICATION".
perl_quote
Convert any Perl value into a source-code fragment that reproduces that value when evaluated in a generated test file.
Arguments
$vAny Perl value. May be undef, a scalar, an arrayref, a Regexp, or a blessed object. All types are handled - undef becomes
'undef', the strings'true'/'false'become the Perl boolean constants!!1/!!0, numbers are unquoted, other strings are single-quoted, arrayrefs recurse, Regexps becomeqr{...}, and anything else (including hashrefs and blessed objects) falls through torender_fallback.
API SPECIFICATION
input
{ v => { type => 'any', optional => 1 } }
output
{ type => 'string' }
The mathematical description is in "FORMAL SPECIFICATION".
NOTES
seed and iterations really should be within config.
SEE ALSO
App::Test::Generator::Template - Template of the file of tests created by
App::Test::GeneratorApp::Test::Generator::SchemaExtractor - Create schemas from Perl programs
Params::Validate::Strict: Schema Definition
Params::Get: Input validation
Return::Set: Output validation
AUTHOR
Nigel Horne, <njh at nigelhorne.com>
Portions of this module's initial design and documentation were created with the assistance of AI.
SUPPORT
This module is provided as-is without any warranty.
You can find documentation for this module with the perldoc command.
perldoc App::Test::Generator
You can also look for information at:
MetaCPAN
GitHub
CPANTS
CPAN Testers' Matrix
CPAN Testers Dependencies
FORMAL SPECIFICATION
This section describes the documented routines in Z notation. It is for readers who want an exact, mathematical statement. You do not need it to use the module.
How to read it: a name ending in ? is an input and a name ending in ! is an output. None of these routines keeps any state between calls, so there is no state schema. quote(v) stands for the result of perl_quote(v).
generate
┌─ Generate ────────────────────────────────────────
│ schema_file? : STRING
│ output_file? : STRING ∪ { undef }
│ test : STRING
│ result! : BOOLEAN
├───────────────────────────────────────────────────
│ pre schema_file? names a readable schema
│ the schema defines function
│ every name in the schema is an identifier
│ post test is a complete Perl test file for function
│ output_file? ≠ undef ⇒ file output_file? holds test
│ output_file? = undef ⇒ test is printed on STDOUT
│ result! = 1
└───────────────────────────────────────────────────
no arguments ⇒ croak 'Usage: generate(...)'
output_file? cannot be written ⇒ croak 'Cannot open ...'
render_fallback
┌─ RenderFallback ──────────────────────────────────
│ v? : VALUE
│ result! : STRING
├───────────────────────────────────────────────────
│ post v? = undef ⇒ result! = 'undef'
│ v? ≠ undef ⇒ result! is one line of Perl source
│ ∧ eval(result!) is a copy of v?
└───────────────────────────────────────────────────
render_hash
┌─ RenderHash ──────────────────────────────────────
│ href? : VALUE
│ result! : STRING
│ kept : ℙ STRING
├───────────────────────────────────────────────────
│ post ref(href?) ≠ 'HASH' ∨ href? = ∅ ⇒ result! = ''
│ ref(href?) = 'HASH' ⇒
│ kept = { k : dom href? | href?(k) is a hashref
│ ∨ href?(k) is a type name }
│ result! has one "'k' => { ... }" entry for each
│ k in kept, in sorted order
└───────────────────────────────────────────────────
render_args_hash
┌─ RenderArgsHash ──────────────────────────────────
│ href? : VALUE
│ result! : STRING
├───────────────────────────────────────────────────
│ post ref(href?) ≠ 'HASH' ∨ href? = ∅ ⇒ result! = ''
│ ref(href?) = 'HASH' ⇒
│ result! = join(', ', ⟨ k : sort(dom href?) •
│ quote(k) . ' => ' . quote(href?(k)) ⟩)
└───────────────────────────────────────────────────
render_arrayref_map
┌─ RenderArrayrefMap ───────────────────────────────
│ href? : VALUE
│ result! : STRING
│ kept : ℙ STRING
├───────────────────────────────────────────────────
│ post href? is false ∨ ref(href?) ≠ 'HASH' ⇒ result! = '()'
│ ref(href?) = 'HASH' ⇒
│ kept = { k : dom href? | ref(href?(k)) = 'ARRAY' }
│ kept = ∅ ⇒ result! = ''
│ result! has one "'k' => [ ... ]" entry for each
│ k in kept, in sorted order
└───────────────────────────────────────────────────
perl_quote
┌─ PerlQuote ───────────────────────────────────────
│ v? : VALUE
│ result! : STRING
├───────────────────────────────────────────────────
│ pre v? is nested no more than 100 levels deep
│ post v? = undef ⇒ result! = 'undef'
│ v? = 'true' ⇒ result! = '!!1'
│ v? = 'false' ⇒ result! = '!!0'
│ v? looks like a number ⇒ result! = v?
│ v? is any other string ⇒ result! = v? in single quotes,
│ with ' and \ escaped
│ ref(v?) = 'ARRAY' ⇒ result! = '[ ' . join(', ',
│ ⟨ e : v? • quote(e) ⟩) . ' ]'
│ ref(v?) = 'Regexp' ⇒ result! = 'qr{...}'
│ otherwise ⇒ result! = render_fallback(v?)
└───────────────────────────────────────────────────
nested more than 100 levels deep ⇒ croak 'structure too deeply nested'
STATE DIAGRAM
generate is a class method and keeps no state. One call runs from start to finish through these stages, and any stage may stop it by croaking.
+-------------+
| START |
+-------------+
|
| generate(schema_file, output_file)
v
+-------------+ no arguments, file not readable
| LOAD SCHEMA |--------------------------------------+
+-------------+ |
| |
| schema parsed |
v |
+-------------+ missing function, bad name, |
| VALIDATE | contradictory input spec |
+-------------+--------------------------------------+
| |
| schema is good |
v v
+-------------+ +-------------+
| RENDER | | CROAKED |
+-------------+ +-------------+
| ^
| test source built |
v |
+-------------+ output_file cannot be opened |
| WRITE |--------------------------------------+
+-------------+
|
| file written, or source printed on STDOUT
v
+-------------+
| DONE |
+-------------+
Side effects: generate writes output_file and prints one line saying so, or prints the test source when no output file is given. The render_* routines and perl_quote write no files and print nothing.
LICENCE AND COPYRIGHT
Copyright 2025-2026 Nigel Horne.
Usage is subject to the terms of GPL2. If you use it, please let me know.