Revision history for App-Test-Generator
0.32 Sun Apr 12 08:16:47 EDT 2026
- Added detect_scattered_failures() root cause detector. Surfaces a
weak-confidence advisory when failures and passes coexist across
2 or more common Perl series with no detectable version cliff or
OS pattern, suggesting flaky tests, optional dependency
differences, or CGI environment assumptions rather than a
compatibility issue. Complements detect_universal_failure() which
handles the opposite case of near-total failure. Confidence is
intentionally set to 0.40 (Weak) since this is a catch-all
signal rather than a precise diagnosis, ensuring it appears below
stronger signals in the root causes table when multiple detectors
fire simultaneously.
- Added detect_universal_failure() root cause detector. Surfaces a
high-confidence warning when failures occur across 3 or more
distinct Perl versions and 2 or more OS types with fewer than
10% passing reports, indicating a likely broken release rather
than a version- or platform-specific compatibility issue. Likely
causes listed in evidence: missing file in tarball, broken
Makefile.PL, or undeclared dependency. Integrated into
detect_root_causes() where it is evaluated first and sorted by
confidence alongside the existing OS, locale, and Perl version
cliff detectors.
- Fixed blib/ paths appearing in coverage table instead of lib/
paths. Devel::Cover instruments blib/ during testing; paths are
now normalised to lib/ for display, with deduplication against
any native lib/ entry.
- Fixed structural coverage percentages in Executive Summary and
Structural Coverage sections showing ~24% instead of ~93%.
_coverage_totals now aggregates from individual own-project files
rather than Devel::Cover's pre-aggregated Total key which
includes all instrumented CPAN dependencies.
- Fixed cyclomatic complexity badge colour and tooltip inverted.
High complexity now correctly shows red/Needs improvement;
low complexity shows green/Good. Second condition also fixed
to use $complexity rather than $score.
0.31 Fri Apr 10 08:07:40 EDT 2026
- Added TER3 (LCSAJ path coverage) column to mutation files table in
the test dashboard index, showing percentage and raw fraction
(e.g. "71.8% (352/491)")
- Added TER1, TER2 and TER3 metrics to per-file mutation report pages,
replacing the plain "Statement" and "Branch" labels with their
formal Test Effectiveness Ratio names
- Renamed the LCSAJ column header in the mutation files table to TER3
- With min set to zero on an integer, sometimes negative or floats could be created - added abs() and int() calls as needed
- Added --generate_mutant_tests=DIR option to generate_index.pl to
produce a timestamped test stub file (t/mutant_YYYYMMDD_HHMMSS.t)
for surviving mutants. High/Medium difficulty survivors get TODO
test stubs; Low difficulty survivors get comment-only hints.
Multiple mutations on the same source line are deduplicated into
one stub listing all variants — one good test kills them all.
Boundary value suggestions are generated for numeric mutations,
clamped and deduplicated for non-negative contexts such as
scalar() and length(). Environment variable hints are added where
the source line references $ENV{...}. The enclosing subroutine
name is shown in each stub for navigation context. File is skipped
if there are no survivors or low-difficulty hints to report.
- LCSAJ path dots on per-file mutation pages are now coloured blue
(covered) or red (not covered), based on whether any line in the
path range was executed during testing. Uncovered paths show
[NOT COVERED] in the hover tooltip. The LCSAJ legend is updated
to explain both colours.
- Replaced TER3-only column in mutation files table with a
TER1 / TER2 / TER3 triple, each component shown as a
colour-coded badge (green/yellow/red). TER1=Statement,
TER2=Branch, TER3=LCSAJ path coverage. Any component
without data shows a grey n/a badge. Column header carries
a tooltip defining all three metrics.
- Added --generate_test=mutant option to generate_index.pl (used
alongside --generate_mutant_tests=DIR). For NUM_BOUNDARY survivors,
attempts to produce a runnable YAML schema file in t/conf/ using
App::Test::Generator::SchemaExtractor rather than a TODO stub.
The schema is augmented with boundary values from the surviving
mutant (the exact boundary value plus one either side) and picked
up automatically by t/fuzz.t on the next test run. Falls back to
a TODO stub if SchemaExtractor fails, the enclosing sub cannot be
found, or confidence in the extracted schema is too low. The
--generate_test option is designed to accept future classes beyond
'mutant'. Updated generate_test_dashboard Step 7 to pass
--generate_test=mutant by default.
- Made output_dir optional in App::Test::Generator::SchemaExtractor
new() -- it is now only required if schema files will be written.
extract_all() gains a no_write option to suppress file output and
return schemas only, for use by callers that want to inspect or
augment schemas before deciding where to write them.
- Added --generate_fuzz flag to generate_index.pl. Scans t/conf/
for existing YAML schema files and writes timestamped augmented
copies (mutant_fuzz_YYYYMMDD_HHMMSS_FUNCTION.yml) with boundary
values from surviving NUM_BOUNDARY mutants merged in. The original
schema is never modified. Augmented schemas are picked up
automatically by t/fuzz.t. Schemas with no matching survivors are
skipped (with a verbose note). Boundary values are merged into
whichever edge key already exists in the schema (edge_case_array
or edge_cases), with deduplication. Schemas already prefixed with
mutant_fuzz_ are skipped to prevent cascading augmentation.
Updated generate_test_dashboard Step 7 to pass --generate_fuzz.
0.30 Thu Apr 2 07:17:16 EDT 2026
Added mutation levels.
Setting to fast will reduce the number of mutants by deduping and removing unnecessary mutants.
See App::Test::Generator::Mutator::_is_redundant_level for a list of those optimised out
Added basic LSCAJ data to the test dashboard.
Added simple security string testing.
Don't push too hard on builtins as they don't have good parameter validation.
Ensure adding only the byte order marker honours $min
More use of _DESCRIPTION
0.29 Thu Feb 26 12:57:59 EST 2026
Added mutation testing
Added guided testing to extract-schemas
Some routines were incorrectly labelled as getter routines
Added more edge cases
Added fallback to extract parameters from classic Perl body styles
Added Type::Param support (https://github.com/nigelhorne/App-Test-Generator/issues/4)
Getter routines take no arguments
Fixed string testing when both min and max are given
Don't give $class or $self as parameters
No input/output no longer croaks, because there are now a few tests that can be run
Add a basic hashref to mandatory args
Use UUID::Tiny and Readonly::Values::Boolean
0.28 Mon Feb 9 19:54:59 EST 2026
Latest test dashboard
Some getter/setter routines were missed
Test getset routines
Added 'isa' test. Tests code dies when passed the wrong type of object
0.27 Thu Jan 29 07:54:01 EST 2026
Fewer false positives for "optional"
Sometimes "new:" was missing
If a return value is 'length($foo)' it's an integer with a minimum value of 0
Some random strings were mistakenly sent as tests that were not in the memberof set
0.26 Mon Jan 12 07:57:54 EST 2026
Added strict_pod mode to SchemaExtractor, which croaks if it finds discrepancies between the POD and code
Improved detection of optional/required arguments
Improved detection of integer output types
Improved POD parser of parameters
Added --dry-run
Improved config validation
Better call of object methods in transforms
Test the sanity of Unicode outputs
Idempotent tests
Prevent silent duplicate method overwrites
Implement confidence_threshold
Refactor _detect_accessor_methods
Only generate one of call_code and position_code
Check for global side effects
Sanity test on DIES
Added timeout tests to ensure the test doesn't hang
0.25 Thu Jan 1 15:41:06 EST 2026
Ternary ? 1 : 0 return is taken to be an indicator of boolean returns
Now parses this pod to know that it's supposed to return a string:
Returns the sanitized HTML string
Use :: to call methods in non-oo modules, rather than ->
Flag when type is set to object by can is not set
0.24 Sun Dec 28 15:10:09 EST 2025
Return chances of false positive file path semantics
If the type of an variable can't be determined, guess at string, but lower the confidence level
Error Message Extraction:
Capture error messages from die/croak/confess
Use them to understand what makes input invalid die "Age must be positive" if $age <= 0;
Store that a negative/zero age is invalid
Use test of scalar(@_) to check for number of optional/mandatory args
Throw a float at a routine that only takes integers - it should error
Added example extraction
Parse POD for example calls and extract valid parameter combinations =head2 SYNOPSIS my $obj = Module->new(foo => 'bar', count => 5);
Store these as known-good test values
Corpus bool testing now uses ok rather than is
Ensure new doesn't refer to other packages
Set the isa property in the new function
0.23 Wed Dec 24 10:48:03 EST 2025
Fix https://www.cpantesters.org/cpan/report/079dba46-d92f-11f0-9642-dde56d8775ea
Don't give false negative that memberof isn't known
SchemaExtractor output files now end in .yml not .yaml
Ensure rand_str isn't called when memberof is set
When test_empty is set, do the right thing for memberof
0.22 Sun Dec 21 20:59:58 EST 2025
Added better API to generate
Added parameter relationship detection (mutually exclusive, required groups, conditional requirements, dependencies, value constraints)
Improved Object Detection
Detect factory methods that return instances
Recognize singleton patterns
Identify when constructor needs specific parameters
Handle inheritance (when parent class new() is needed)
Better Default Value Extraction
$param = $param || 'default_value'
$param //= 'default_value'
$param = defined $param ? $param : 'default'
Expanded _detect_list_context to
Better distinguish scalar vs list returns
Detect void context methods (those that modify state)
Recognize chaining patterns more reliably Identify error return conventions (undef on failure, etc.)
Add confidence scoring transparency, explain WHY confidence is what it is:
_analysis: confidence_factors:
- "POD documentation present (+30)"
- "Type validation in code (+20)"
- "No constraint information (-10)"
Add support for:
- Signatures (sub foo($x, $y = 5) { })
- Postfix dereferencing ($array->@*)
- Subroutine attributes (sub foo :Returns(Int) { })
- Field declarations (Perl 5.38+)
Added a new section to generated YAML:
yamltest_hints: boundary_values: [0, 1, 100, 255]
detected from code invalid_inputs: ['', undef, -1]
from validation checks equivalence_classes: []
0.21 Sun Dec 14 08:07:09 EST 2025
Schemaextractor: don't put the package name as the argument
Validate config settings better
Fix max string testing with non-ASCII characters
Changed rand_str to be a unified generator that randomly produces codepoint strings,
grapheme clusters, ZWJ emoji sequences, or aggressive Unicode fuzz strings
Schemaextractor: Added advanced type detection for DateTime objects, file handles, coderefs, and enum validation patterns
Added enum as a synonym of memberof
Added tests for unix_timestamp semantic type
0.20 Fri Dec 5 07:53:43 EST 2025
Added the --version flag to fuzz-harness-generator
Ensure the max value of string is honoured better
Fix array context detection to only match return statements
Improve chances of detecting a boolean output
Make the list context detection more specific
0.19 Wed Nov 26 07:50:22 EST 2025
Fixed the loop iterating over mandatory args
Ensure mandatory_args honours matches
0.18 Tue Nov 25 11:58:12 EST 2025
Removed one place sending '' if test_empty was disabled
Don't send long strings if matches is set
Fixed handling of position code with more than argument
Renamed the sample module
0.17 Tue Nov 25 10:24:36 EST 2025
Improved TEST_VERBOSE output for Corpus tests
Make the strings more random
Added test_non_ascii (default 1) to the configuration matrix
Removed all legacy conf code
Re-ordered the POD
Don't give 42 if max < 42
Added App::Test::Generator::SchemaExtractor
0.16 Thu Nov 20 08:38:17 EST 2025
Use Data::Random::String to generate the string
Use Data::Random::Structure to generate references to hashes and arrays
Remove legacy support for Perl config files
Handle the case when all inputs should cause a die(),
allows App::Test::Generator to sanity test itself
If something is expected to die, don't look at its return code
More boolean edge cases
Don't pass _NAME to validate_strict
When a routine dies, it shouldn't return anything
Begin to use semantic types
Use Test::LectroTest to create more tests
Added custom properties
Separated the Template into its own class to ease maintenance
Use reusable functions to generate cases
Rand_set returns and array, so look at the first element
0.15 Tue Nov 11 19:30:18 EST 2025
Documented how to Fuzz test a CPAN module
populate_positions - don't attempt to deref a scalar
carp when undef is a corpus test case, but test_undefs is not set
Improved handling of cases array for WARNS and DIES
Improved handling of validating position settings in config files
Croak if the schema_file can't be opened
Close stdin and use /dev/null
Fixed logic in testing short strings when min >= 2
0.14 Fri Nov 7 20:26:32 EST 2025
Fixed logic when neither memberof nor notmemberof were defined,
which could cause an empty notmemberof array to be created
Transform: better number of items for arrayrefs
At least one of module and function must be defined
Don't send wrong data types in transform testing built ins
Document the edge_cases_array and ensure all tests in that array are run
Better handing of "input: undef" in the config file
Fix handling of matches in output schemas
0.13 Tue Nov 4 11:52:49 EST 2025
Added better documentation for transforms
transform: filled in more types for the foundation set of data
Added "value" to the transform keywords
Refactored and enabled the code to validate that $module exists and is installed
Added deprecation notice about loading configs from perl code
Allow yes/no as booleans in schema config settings
Fix sprintf issue when the text contains a % sign
0.12 Sun Nov 2 19:38:10 EST 2025
undef can now be used to indicate no input or output
Sending with no args if now configured using "test_undef" rather than "test_empty"
Use Data::Random::String::Matches to create random string that matches a regex
Flag keywords that are yet to be handled
$module can now be set to "builtin"
Initial basic support for positional arguments and transforms
0.11 Sun Oct 26 14:39:19 EDT 2025
Only add to candidate_bad those test cases which are requested
Better matches and nomatch tests
Only load JSON module when dedup is enabled
Added the -o and -r options to bin/fuzz-harness-generator
0.10 Wed Oct 22 09:23:38 EDT 2025
Send wrong data types - should die, e.g. string to an integer field
Fix syntax error when Math::Simple is installed
Added test_empty config (default is on)
0.09 Sun Oct 19 21:14:29 EDT 2025
Better handling of false/true in config files
Better handling of regex into the input structure
0.08 Thu Oct 16 20:44:22 EDT 2025
Allow $new to be set but not defined, this will call new() with no arguments
Started to add support for "nomatch"
Allow specs that specify no input if they have output
If a routine takes no input, it won't die if it has no input
0.07 Tue Oct 14 16:36:33 EDT 2025
Bump minimum versions
Added float that was missing some places
Use Data::Random
Fix GitHub#3
Show how to use this module to automatically schedule random tests
Test for code doing if($string) rather than if(defined($string)) which is confused if $string is '0'
Allow the special word "undef" in the YAML specification for output
0.06 Fri Oct 10 08:55:36 EDT 2025
Allow real config files to be read
Validate the configuration file
0.05 Thu Oct 9 18:23:03 EDT 2025
Removed duplicate regex candidate table
Validate that the corpus inputs are arrayrefs
Only load Class::Simple when needed
Use rel2abs
Load the configuration file in a (slightly) safer way
The test code is now in a template toolkit
Improved random test generator, more knowledgable about min/max, memberof and matches
0.04 Wed Oct 8 08:39:49 EDT 2025
Add fuzzy regex generator
Do basic hard-coded tests where possible, to get it started
Allow pathnames in the module name
Added qwrap - GitHub#1 - thanks to neo1ite
Don't try to fuzz input if no %input is given
Added %config - GitHub#2
Generate tests for routines that take one unnamed parameter - GitHub#2
Added fallback for perl_quote for hashes and objects
0.03 Mon Sep 29 18:18:36 EDT 2025
If minimum is not set, verify 0 or empty fields are allowable
Added the testing dashboard
Fixed handling of memberof in input/output array creation
If TEST_VERBOSE is set, print the generated dataset when running it
Always ensure mandatory strings are passed when testing other arguments
rand_int and rand_numb now also sometimes return very large and very small numbers
Put utf-8 and NUL bytes into strings
0.02 Sun Sep 28 09:03:49 EDT 2025
Use gtar on OS/X to generate the distro
Added edge case test generator for booleans and memberof
0.01 Sun Sep 28 08:43:35 EDT 2025
First draft