Revision history for App-Test-Generator
0.40 Thu Jun 25 20:28:59 EDT 2026
[Bug fixes]
- Fix bin/generate-test-dashboard passing --changed_only to
test-generator-mutate without a --base_sha, causing the diff to
cover only HEAD~1. With dashboard.yml's concurrency cancel-in-progress
policy, coalesced CI runs left stale survivor entries for any file not
touched in the final commit of a batch; those entries were never
refreshed by subsequent runs. Now computes BASE_SHA by skipping the
workflow's own bookkeeping commits (same logic as mutate.yml), so the
diff always covers everything since mutation.json was last meaningfully
regenerated.
- Fix bin/test-generator-mutate setting $ENV{LCSAJ_TARGETS} to the
mutation-scoped file list before running the baseline system('prove')
check. This caused t/LCSAJ-Runtime.t (which calls DB::DB() directly
against its own file) to fail spuriously whenever --file or
--changed_only selected a subset of lib files, aborting the entire
mutation run before any mutants were tested. LCSAJ_TARGETS is now
assigned after baseline verification, scoped only to the mutation runs.
- Fix t/LCSAJ-Runtime.t's hit-recording subtest assuming %TARGET is
empty. During mutation testing LCSAJ_TARGETS is intentionally scoped
to the file under mutation (never a test file), so %TARGET is
non-empty and the subtest's direct DB::DB() call was filtered out,
making the 'hit recorded' assertion fail for every per-mutant prove
run. This produced a fake 100% kill rate that invalidated all mutation
results. Fixed by using local %TARGET = () within that subtest.
- Fix t/function.t's BEGIN-time LCSAJ_TARGETS subtest spawning its
child perl process via qx{}/backticks with a shell-quoted -e
argument (single quotes). qx{} always goes through a shell, and
cmd.exe on Windows does not treat single quotes as a quoting
character, so the child process failed to parse its own -e
script ("Can't find string terminator"). Switched to list-form
system() under Capture::Tiny::capture_merged(), which passes the
-e argument directly with no shell involved on any platform.
- Fix LCSAJ::_save_lcsaj() building an output directory path by
joining $dir with $file's full path when $file has no 'lib/'
segment to strip (e.g. a File::Temp tempfile). On Windows this
embedded a second drive letter mid-path and made
File::Path::make_path fail with "Invalid argument"; on other
platforms it silently created a deeply nested mirror of $file's
absolute path under $dir. Now falls back to the basename when
the stripped path is still absolute.
- Removed Test::Needs gates that never reflect a genuine dependency:
t/params_validate.t, t/moosex_params_validate.t (Params::Validate
and MooseX::Params::Validate/MooseX::Types::Moose appear only in
heredoc fixture text parsed by PPI/Safe::reval, never truly loaded),
t/params_validate_strict.t (already a hard PREREQ_PM), and the
HTML::Genealogy::Map gate in t/fuzz.t (leftover from deleted
t/conf/html_genealogy_map.{conf,yml} fixtures). The
MooseX::Params::Validate fix restores real extraction coverage
that was previously silently skipped on any machine without Moose
installed.
- Added a Test::Needs gate for Type::Params/Types::Common to the
signature_for subtest in t/integration.t: that fixture is compiled
by a genuinely spawned perl -T subprocess, which
Test::Without::Module cannot fake "missing" for, so the dependency
must be a hard skip rather than left to croak.
- Fix Planner::Fixture::plan() comparing $isolation->{$method}
directly against the 'shared_fixture' string. Planner::Isolation::
plan() actually returns a per-method hashref with a 'fixture' key
(plus optional env/filesystem/time/network keys), not a bare mode
string, so the comparison always stringified a hash reference and
was always false. In practice this meant every method planned by
Planner::build_plan() silently got 'new_per_test' fixture mode,
even purity-'pure' methods that should have gotten a shared
fixture. Each module's own unit tests passed because they fed
Fixture::plan() a fabricated flat-string isolation map that never
matched Isolation::plan()'s real output shape; only composing the
two through build_plan() in a new end-to-end test caught the
mismatch. Fixed to read $isolation->{$method}{fixture}; updated
Fixture.pm's POD and t/Planner-Fixture.t, t/Planner-submodules_unit.t,
and t/function.t's fixtures to the real per-method hashref shape.
- Fix mutation.json incorrectly added to .gitignore while the file is
tracked by git; removed the .gitignore entry and untracked the file
with git rm --cached so it is no longer committed.
- Fix mutate.yml (deployed to dependent repos) still using floating
shogo82148/actions-setup-perl@v1 after dashboard.yml was pinned;
both deployed workflow files now pin to the same version.
- Fix POD-embedded CI workflow template referencing stale action
versions (actions/checkout@v5, actions-setup-perl@v1) after
dashboard.yml was updated; template now matches dashboard.yml.
- Fix README.md SEE ALSO link label still saying "Test Coverage Report"
after the POD was updated to "Test Dashboard".
- Fix dashboard.yml "Publish test dashboard" step running unguarded on
pull_request events, publishing PR-derived coverage output to the
public gh-pages site; now skipped on pull_request like the other
commit steps.
- Fix dashboard.yml "Save updated mutation results" guard
(if: github.event_name == 'workflow_run') discarding the freshly
regenerated mutation.json on plain push runs; generate-test-dashboard
regenerates mutation.json on every run regardless of trigger, so the
guard now matches its siblings (if: github.event_name !=
'pull_request').
- Fix dashboard.yml silently skipping the coverage snapshot when
cover_html/cover.json is missing, masking real dashboard-generation
failures; the step now emits an ::error:: annotation, dumps
cover_db/cover_html/coverage for diagnosis, and fails the job.
- Fix mutate.yml "Find last code commit" silently falling back to
HEAD~1 when no base commit could be parsed from git log, narrowing
the mutation diff without any visible indication; now emits an
::warning:: annotation when BASE is empty.
- Fix dashboard.yml and mutate.yml commit steps calling a shared
.github/scripts/commit-if-changed.sh helper; dependent repos only
receive the two workflow YAML files when copying them in (see
"now portable to any CPAN module" below), so the script was missing
on every consumer and CI failed with "No such file or directory".
Commit/push logic is now inlined directly in each step.
- Fix extract-schemas _load_target_module() loading the target module
via eval "require $package" (string eval); the package name is now
validated against Perl's package-name grammar and the module is
loaded by converting it to a file path and calling require on that,
so it is never compiled as arbitrary Perl source.
- Fix test-generator-mutate --changed_only interpolating --base_sha
into a backtick-executed git diff command; --base_sha is now
validated against a restrictive character class and git diff is
invoked via list-form open() instead of the shell.
- Fix test-generator-index _mutant_file_report() computing a
slash-sanitised $filename that was never used, leaving the report
path built from the raw, unsanitised $file; it now rejects any '..'
path segment in $file and verifies the resolved output path stays
under the report directory before writing.
- Fix extract-schemas _load_target_module() losing the drive letter
when reconstructing the lib-dir candidate with File::Spec->catdir(),
so the @INC-walking loop never found a 'lib' directory on Windows
when the input file and the cwd were on different drives; now uses
catpath() to keep the volume attached.
- Fix SchemaExtractor _infer_type_from_expression() and
_parse_modern_signature() using hand-rolled brace/bracket depth
counters to split comma-separated lists, which mishandled nested
brackets (e.g. a signature default value containing a hashref
literal could be split on the comma inside the hashref); both now
use Text::Balanced::extract_bracketed() to skip over nested
structures correctly.
- Fix Mutator.pm's _dedup_mutants(), _is_redundant_mutation(), and
apply_mutant() reaching into Mutant objects via direct hash access
(e.g. $m->{line}), bypassing the class's own accessor methods; all
six call sites now use the Mutant accessors ($m->line, $m->original,
$m->transform, etc).
- Fix Devel::App::Test::Generator::LCSAJ::Runtime's DB::DB debugger
hook calling abs_path() (a filesystem stat) on every single
statement executed under the debugger; resolved paths are now
memoised per raw $file in %NORM_CACHE.
- Fix lib/App/Test/Generator/Sample/Module.pm declaring
package Test::App::Generator::Sample::Module (reversed word order)
instead of App::Test::Generator::Sample::Module, so the package
name did not match its location under
lib/App/Test/Generator/Sample/; renamed throughout the file.
- Fix Generator.pm SYNOPSIS extract-schemas example passing two
positional arguments (bin/extract-schemas lib/Sample/Module.pm)
instead of one, referencing a nonexistent lib/Sample/Module.pm path,
and naming the generated schema file schemas/greet.yaml when
extract-schemas actually writes schemas/greet.yml; corrected all
three.
- Fix lib/App/Test/Generator/Exporter/YAML.pm having no package
statement (so it loaded into main::), no input validation before
YAML::XS::DumpFile(), and no test coverage; added the package
statement, Params::Validate::Strict validation of plan/file
(rejecting non-hashref plans and missing/empty file paths), and
t/Exporter-YAML.t.
- Fix generate()'s _assert_identifier() check on the function name
rejecting fully-qualified sub names such as DB::DB, breaking
t/app.t's self-test sweep against
Devel::App::Test::Generator::LCSAJ::Runtime.pm (DB::DB is a Perl
debugger hook, always installed into the DB:: package regardless
of its source package); the function-name check now allows "::"
separators, same as the existing module-name check.
- Fix SchemaExtractor's heuristic numeric-type inference missing
parameters that are only ever validated via an explicit
looks_like_number($param) call rather than direct arithmetic
adjacency (e.g. a parameter used as looks_like_number($b) in a
guard clause and only later folded into arithmetic via
$a + ($b // 0), where $b never sits next to an operator itself);
such parameters fell back to type 'string', so fuzz-generated
non-numeric inputs caused generated tests to fail against code
that legitimately dies on non-numeric input. An explicit
looks_like_number($param) call in the source is now treated as a
direct numeric-type assertion, checked before the existing
arithmetic-operator and comparison heuristics.
- Fix _compile_signature_isolated()'s "fast path" Safe compartment,
tried before falling back to the subprocess unconditionally;
Type::Params/Types::Common pull in XS modules and Safe cannot host
XS/dynamic loading, so the compartment never succeeded for any real
signature_for() declaration and was dead code giving a false
impression of sandboxing. Removed; the subprocess path (gated on
allow_signature_exec => 1) is now the only path.
- Fix Generator.pm splicing module/function/transform/field names
unescaped into generated test source; added _assert_identifier()
and applied it before every such splice point so a name that is
not identifier-shaped now croaks instead of producing a test file
with injected code.
- Fix extract-schemas2 calling Planner->new without the required
package argument, crashing on every invocation; package is now
passed through from the extracted schema.
- Fix Template.pm _dedup_cases() always being a no-op: a return
inside the eval{} block returned from the eval, not from
_dedup_cases, so the deduplicated result was discarded and the
unduplicated $cases was always returned to the caller.
- Fix SchemaExtractor _detect_dependencies() relationship detection
losing pos() state on a shared $code string across nested global
matches; affected loops now operate on independent copies/resets.
- Fix SchemaExtractor numeric boundary-value hints double-counting
values already present from a previous hint pass; boundary values
are now added through a seen-value set, so re-running hint
generation on the same schema no longer duplicates values.
- Fix SchemaExtractor _detect_external_object_dependency() not
resetting pos($method_body) before its global match, so a prior
match elsewhere on the same string could cause this detector to
start scanning mid-string and miss leading dependencies; pos() is
now explicitly reset to 0 first.
- Fix CoverageGuidedFuzzer calling target_sub with no time bound, so
a target that hangs (e.g. on an unexpected infinite-loop input)
hung the whole fuzz run; calls are now wrapped in alarm()-based
timeout (default 5s, configurable via timeout => N, 0 disables).
- Fix LCSAJ.pm's CFG builder connecting every branch point only to a
newly-created true-block successor, leaving the false-arm block
disconnected from the frontier and silently dropped as an empty
leaf with no lines or edges; both true and false successors are
now wired into the frontier so subsequent statements are recorded
against both arms.
- Fix Mutator generate_mutants() never calling each mutation
strategy's applies_to() pre-filter before mutate(), and never
populating the context/line_content fields on the Mutant objects
it constructs; both are now wired through.
- Fix Analyzer::SideEffect.pm and Analyzer::Complexity.pm counting
IO/exec keywords, branch/logic/exception keywords, and the
literal ? character when they merely appeared inside a string
literal or comment (e.g. "Are you sure?" or a # comment containing
"die"); source is now stripped of string and comment content
before these keyword/operator counts are taken.
- Fix Analyzer::Return.pm's \$self(?!->) returns_self pattern
matching the \$self prefix of any longer variable name (e.g.
\$self_backup, \$selfish); added a \b word boundary. Also fixed
all three return-pattern matches running as a single non-/g match,
so a method with several qualifying return statements only ever
contributed one evidence entry instead of one per occurrence.
- Fix Mutation::BooleanNegation and Mutation::ReturnUndef mutating
only the first PPI token of a multi-token return expression (e.g.
$self->{value}), since $ret->schild(1) captures just the leading
Symbol token of a chained/dereferencing expression; this produced
broken mutants like 'return !($self)->{value};' and
'return undef->{value};' that die at runtime rather than testing
the intended boolean/undef substitution. Both now resolve the
full expression span (up to a trailing postfix conditional/loop
modifier or the statement terminator) and wrap/replace it as a
whole.
- Fix SchemaExtractor _analyze_relationships() extracting parameter
names with a hardcoded my (...) = @_ regex only, so shift-style
(my $x = shift) and modern-signature (sub foo($self, $x)) methods
never had their parameter relationships (mutually-exclusive,
required-group, dependency, etc.) analysed; parameter extraction
now reuses _extract_parameters_from_signature(), which already
supports all four parameter-declaration styles.
- Fix Analyzer::ReturnMeta.pm's stability bonus for boolean returns
being undocumented in POD: since stability_score starts at 100 and
is clamped to [0, 100], the bonus is a no-op unless an earlier
penalty already reduced the score, which POD readers had no way of
knowing. Documented the no-op-in-the-common-case behaviour.
- Fix Planner::Mock.pm plan() silently dropping the capture_io mock
for any method that is both calls_external and performs_io, since
the if/elsif gave mock_system precedence; such a method now gets
an arrayref of both mock strategies instead of losing one.
- Fix CoverageGuidedFuzzer.pm _run_with_cover() walking the full
Devel::Cover state twice per fuzz iteration (once for "before",
once for "after"); coverage state only grows, so this iteration's
"before" is now the cached "after" snapshot from the previous
iteration, halving the per-iteration Devel::Cover walk count.
- Fix CoverageGuidedFuzzer.pm _validate_value() matching a schema-
supplied 'matches' regex against fuzzer-generated input with no
bound on catastrophic backtracking; the match is now wrapped in
the same alarm()-based timeout already used to bound target_sub
calls, and a timeout is treated as a non-match.
- Fix SchemaExtractor.pm's TODO POD section claiming union-type
parsing (e.g. scalar | scalarref) in =head4 Input specs was
unimplemented; _map_formal_input_type already handles it. Narrowed
the TODO to the part that is genuinely still unimplemented (the
enum/memberof constraint synonym).
- Fix TestStrategy.pm's getset-accessor branch picking an arbitrary,
hash-order-dependent input parameter when more than one candidate
was present; candidates are now sorted by their schema position
field first, making the choice deterministic.
- Fix LCSAJ::Coverage.pm's custom "Cannot read $file: $!" / "Cannot
write coverage output to $out_file: $!" croak messages being
unreachable dead code under "use autodie qw(:all)" (autodie's
open() never returns false, so "open(...) or croak(...)" never
fires); autodie is now disabled locally for these two open calls
so the documented custom messages are actually produced on failure.
[Enhancements]
- Add cross-module end-to-end subtests to t/integration.t:
Planner::build_plan()'s full five-subsystem composition (verifying
the documented TestStrategy/Mock/Isolation/Fixture/Grouping
responsibility split with real side-effect/dependency metadata,
and that build_plan() calls each subsystem exactly once via
Test::Mockingbird::spy); a SchemaExtractor -> Exporter::YAML ->
disk -> Generator round trip; independent-instance concurrency
checks for Mutator and CoverageGuidedFuzzer (interleaved calls on
separate instances must not leak state); and a Test::Without::
Module-based fallback check proving SchemaExtractor's
signature_for() extraction is unaffected by BSD::Resource being
unavailable. Test::Returns' returns_ok() is used to validate
build_plan()'s output against its documented API specification.
- Pin shogo82148/actions-setup-perl to immutable commit SHA
(a198315 / v1.41.1) in dashboard.yml, mutate.yml, and the
POD-embedded CI template; actions/checkout similarly pinned to
df4cb1c (v6) in the POD template.
- dashboard.yml and mutate.yml commit/push steps now do a
git pull --rebase before pushing, reducing non-fast-forward
rejections when mutate.yml's commit and dashboard.yml's
workflow_run-triggered commit land close together.
- Add destructive/hostile subtests to t/edge_cases.t, scoped to
angles not already covered by t/function.t or t/Model-Method*.t:
Planner::Isolation::plan() with an undef $schema (only $strategy is
hashref-validated; $schema degrades safely to the isolated_block
fixture); Model::Method::evidence_ref() external-mutation/aliasing
abuse (a forged entry bypassing add_evidence()'s category/signal
validation is still summed by resolve_confidence()); evidence()'s
documented list-vs-scalar context difference; an unlocalized-$_
confirmation for resolve_confidence()'s postfix for loop;
Analyzer::SideEffect::analyze() with an arrayref method body
(stringifies harmlessly) and a typeglob in place of $method (dies
with Perl's native "Not a HASH reference"); Exporter::YAML::export()
with a circular-reference plan hashref (YAML::XS handles it via
anchors/aliases, confirmed not to hang); TestStrategy::generate_plan()
with an undef vs. non-hashref per-method schema entry (the former
degrades to basic_test, the latter dies under strict refs); and an
end-to-end Generator::generate() injection test confirming a
statement-injection-shaped function name is rejected by
_assert_identifier() before any output file is written.
0.39 Sun May 24 17:43:17 EDT 2026
[Bug fixes]
- Fix mutation dashboard Total column excluding skipped mutants; Total now
equals Killed + Survivors + Skipped. Score remains based on Killed / (Killed
+ Survivors) so skipped mutants do not penalise the mutation score.
- Fix extract-schemas --strict-pod false positive when a method uses
Params::Get::get_params('key', \@_) to extract arguments; the key name
is now recognised as a code parameter, resolving POD/code disagreement.
- Fix _extract_defaults_from_code incorrectly marking a parameter as
optional when its only assignment is a dereference or method call
(e.g. my $x = $params->{x}); such expressions are not default values.
- Fix extract-schemas inferring output type as string instead of hashref
when a method has a =head4 Output formal spec; the spec type now takes
precedence over heuristic code analysis (the string came from scalar
being mapped to string when it wasn't in the valid-types list).
- Fix extract-schemas inferring output type as string instead of array when
a =head4 Output block uses list notation (...); the output-type validator
did not include 'array' in its allowed-types set and silently fell back
to string.
- Fix SchemaExtractor emitting both 'enum' and 'memberof' for the same
parameter when a regex-based enum is detected; the two fields are mutually
exclusive and Generator.pm now rejects schemas that contain both.
SchemaExtractor now only sets memberof when enum is not already being
output in the cleaned schema.
- Fix SchemaExtractor falsely inferring type 'number' for parameters whose
code uses the defined-or operator (//) — e.g. $text // '' was mistaken
for division because the numeric-operator pattern included bare / without
excluding //. Both heuristic-inference sites now use \/(?!\/) to require
that the / is not followed by a second /.
[Enhancements]
- SchemaExtractor now parses =head3 Input and =head4 Input formal spec
blocks, mirroring the existing =head4 Output support. Positional array
format ([ {type=>'...'}, ... ]) and named hash format
({ name => {type=>'...'}, ... }) are both supported. The spec takes
highest priority, overriding all heuristic type inference. Union types
such as 'scalar | scalarref' are resolved to the canonical ATG type
(both scalar and scalarref map to string).
- Add t/app.t: ATG self-test that runs every lib/**/*.pm through
SchemaExtractor (strict_pod=fatal), generates a fuzz harness for each
extracted schema via App::Test::Generator, and runs it through prove.
Temp dirs containing the intermediate schema YAML and generated .t files
are kept on failure for diagnosis. Runs as an extended test.
- Support array-returning methods end-to-end, enabled by Test::Returns 0.03
which accepts type => 'array' as a synonym for type => 'arrayref':
extract-schemas now emits output.type: array for list-returning methods,
the output-type validator accepts 'array' as a valid type, and generated
tests capture the result in list context (my @_r = CALL; $result = \@_r)
so that returns_ok can validate it as an arrayref.
- Fix strict_pod=fatal incorrectly flagging methods with no POD at all;
the check now only runs when POD is present (validates accuracy of
existing documentation, not completeness).
- Fix strict_pod=fatal false positive: $class and $self are always treated
as invocants and never reported as undocumented code parameters,
regardless of method name.
- Fix _extract_pod_before accumulating multiple adjacent POD blocks; it
now stops after the first pod token so class-level POD separated by
=cut does not bleed into a method's specific POD context.
- Fix generated tests dying without done_testing() when an input parameter
has type arrayref or array; both types are now handled in the mandatory-
argument setup block in Template.pm.
- Fix extract-schemas creating spurious schema files for Perl special blocks
(BEGIN, END, DESTROY, AUTOLOAD, CHECK, INIT, UNITCHECK); these are now
skipped in _find_methods.
- Fix extract-schemas failing to emit new: ~ for instance methods that use
the my $self = $_[0] direct-index invocant style and call other instance
methods via $self->method() but do not dereference $self directly;
_detect_instance_method now recognises $_[0] as explicit_self (high
confidence), and _needs_object_instantiation now promotes methods that
call instance methods on $self even when no hash/array dereference is
present.
- Fix --strict-pod false positive for methods using the my $self = $_[0]
calling style: add direct-index style recognition to
_extract_parameters_from_signature and guard the fallback extractor in
_extract_defaults_from_code so that inner closure parameters
(my ($x, $y) = @_ inside a sub ref) are not mistaken for the outer
method's parameters.
[Bug fixes]
- Fix Mutation::NumericBoundary's conditional-context detection
(used for Mutator's fast-mode dedup) always evaluating false: PPI
wraps a condition's content in PPI::Statement::Expression, so an
operator's immediate parent is never literally
PPI::Structure::Condition. Operators inside if/unless/while/until
conditions were always tagged 'expression'. Now uses the same
ancestor-walking _in_conditional() helper already shared by
BooleanNegation and ReturnUndef.
- Fix Analyzer::SideEffect.pm's mutates_self and mutates_globals
detection ($self->{field} = ... assignment and %ENV/%SIG/@ARGV
mutation) matching against the raw method body instead of the
comment/string-stripped $code_only, so a field-assignment-like
fragment inside a string literal or comment could be mistaken for
an actual mutation. Both checks now match against $code_only, same
as the keyword/operator counts already fixed in a prior release.
- Fix Emitter::Perl.pm's _emit_method_tests() having no dispatch
branch for the boundary_tests plan flag: TestStrategy.pm sets
$plan{boundary_tests} when a method's schema carries non-empty
_yamltest_hints, but the corresponding $TEST_BOUNDARY constant was
defined and never read, so methods planned for boundary testing
silently got zero generated test code for it. Added the dispatch
line plus a new _emit_boundary_test() that emits one smoke-test
block per boundary_values/invalid_inputs hint value.
- Fix TestStrategy.pm's generate_plan() extracting side_effects and
dependencies from each method's schema _analysis but never using
them, while the module's own DESCRIPTION claimed the plan was
based on them; verified empirically that schema side-effect data
has zero effect on the generated plan. Removed the dead
extraction and corrected DESCRIPTION to attribute side-effect- and
dependency-driven planning to Planner::Mock and Planner::Isolation,
where it actually happens.
- Fix SchemaExtractor.pm's include_private POD describing the
always-included _new/_init/_build method-name override as an
exact-name match; _find_methods actually does a prefix match
(/^_(new|init|build)/), so e.g. _build_attribute and _init_logger
are also force-included, matching common Moose builder/
initializer naming conventions. Confirmed via git log -L that the
prefix behaviour is original and deliberate; POD corrected to
match.
- Fix Generator.pm's render_hash() POD describing $href's values as
always hashrefs; a scalar value that is a recognised type string
is silently expanded to { type => $value } before being skipped-
with-a-warning, which the POD did not mention.
- Fix Generator.pm's _perl_quote() POD omitting that the strings
'true'/'false' are special-cased to the Perl boolean constants
!!1/!!0 rather than being single-quoted like other strings.
- Fix Analyzer::Complexity.pm and Analyzer::Return.pm POD describing
their $method argument as an App::Test::Generator::Model::Method
object; SchemaExtractor's actual callers pass a plain hashref with
a body/source key, not a Model::Method instance. POD corrected to
describe the real calling convention.
- Fix README.md's extract-schemas usage example passing two
positional arguments and referencing a nonexistent
lib/Sample/Module.pm with a .yaml extension; corrected to the real
invocation (extract-schemas lib/App/Test/Generator/Sample/Module.pm
&& fuzz-harness-generator -r schemas/greet.yml) and updated its
GitHub Actions example to the same pinned action SHAs already used
in dashboard.yml/mutate.yml.
[Enhancements]
- Add comprehensive black-box subtests to t/unit.t validating the
public, POD-documented API of every .pm file under lib/ (all 27
modules), written strictly against each module's documented
Arguments/Returns contract rather than incidental implementation
behaviour. Where a module's existing dedicated test file
(e.g. t/Planner-Isolation.t, t/TestStrategy.t,
t/Template_unit.t) already exercised its public API exhaustively
in equivalent black-box style, no duplicate coverage was added to
t/unit.t.
- Add missing public-API POD (Purpose, Arguments, Returns, API
specification) to every previously undocumented public method in
Model::Method.pm, which had none beyond a VERSION section; to
Template.pm's get_data_section(); and to Planner.pm's build_plan(),
which previously carried only an internal TODO-style comment
despite being a public method.
- Add Notes sections to Mutation::BooleanNegation, ::ConditionalInversion,
::NumericBoundary, and ::ReturnUndef documenting the context and
line_content fields each mutant carries for Mutator's fast-mode
dedup, and to CoverageGuidedFuzzer.pm documenting that a
target_sub die is only recorded as a bug when the triggering input
is schema-valid.
0.38 Mon May 18 21:19:40 EDT 2026
[Bug fixes]
- Fix missing TER1/TER2/TER3 column in mutation dashboard
- remove duplicate _lcsaj_coverage_for_file calls
- move skipped-annotation note inside summary div
- Fix MUTANT_SKIP_BEGIN/END regex to match only lines where the annotation is the entire content,
preventing false positives in comments and POD
0.37 Mon May 18 14:58:48 EDT 2026
[Enhancements]
- Add MUTANT_SKIP_BEGIN / MUTANT_SKIP_END source annotations to exclude
lines from mutation testing; mismatched markers are fatal. Skipped
line counts appear in the per-file mutation report and summary table.
[Bug fixes]
- Fix fuzz schema generation looking in xt/conf instead of t/conf.
- Fix mutate.yml for external repos: add commit step so mutation.json
is persisted after each run, enabling the dashboard workflow_run
trigger to pick up fresh results.
- Fix BSD::Resource dependency failing on Windows; guard with OS check.
0.36 Sat May 9 18:54:04 EDT 2026
[Bug fixes]
- Fix uninitialized $version warning in CPAN Testers "no failures"
message when the CPAN API returns a 404 (no release data available).
- Fix "Can't open lib/App/Test/Generator.pm" error when test-generator-index
is run from a repository other than ATG itself; ATG version is now
found by searching @INC with fallback to the configured module_file path.
0.35 Sat May 9 09:40:08 EDT 2026
[Enhancements]
- Dashboard footer now shows the App::Test::Generator version used
to generate it, linking to the MetaCPAN distribution page.
- Executive Summary wording now varies by mutation score: positive
framing for high scores, an actionable hint for low scores, and a
clear message when no mutation data is available yet. Thresholds
use med_threshold and low_threshold from %config rather than
magic numbers.
- Coverage dashboard per-file HTML links now work correctly; Devel::Cover
instrumentation restored to use cover -test for accurate per-file
HTML generation.
- Dashboard file links now correctly target blib-lib-* filenames as
generated by Devel::Cover, fixing persistent 404 errors on all
per-file coverage pages.
- generate-test-dashboard now derives the Devel::Cover -select pattern
dynamically from GITHUB_REPOSITORY, making the script portable across
CPAN distributions without hardcoded module paths.
- Fuzz schema generation now correctly looks in t/conf rather than
xt/conf for existing schemas to augment.
- Redundant exclusion of mutant_*.t from prove invocation removed;
mutant stubs are in xt/ and were never matched by the t/ find anyway.
[Bug fixes]
- Fix https://www.cpantesters.org/cpan/report/04c7279a-476f-11f1-bf55-cb595875c975
Make t/type_params.t an extended test
0.34 Sun May 3 10:30:24 EDT 2026
[Bug fixes]
- Emitter::Perl: _emit_void_test() generated ok(!defined $result || 1, ...)
which is a tautology and always passes regardless of return value.
Fixed to ok(!defined $result, '$method returns nothing (void)').
- Generator: schema files named after Perl builtins (e.g. abs.yml) caused
generated tests to emit "use abs" which fails to compile. Added
_is_perl_builtin() guard to prevent builtin names being used as module
names in use_ok() calls.
- bin/test-generator-index: mutant stub generator incorrectly called
new_ok() for class methods such as App::Test::Generator::generate().
Added _is_class_method() helper to detect $class vs $self and emit
the correct calling convention in generated stubs.
- bin/test-generator-index: CPAN Testers failure table showed all
expected NA rows for Perl < 5.036 as noise. Rows below the detected
Perl version cliff are now omitted with a count notice, leaving only
unexpected failures visible.
- CoverageGuidedFuzzer: _load_json_module() used require $mod where $mod
contained '::' which Perl does not convert to path separators for
variable require. Fixed with s{::}{/}g conversion before require.
[Enhancements]
- Comprehensive test suite added across all modules:
- Function tests (white-box) for all private helpers
- Black-box unit tests for every public method against POD API spec
- Integration tests covering 10 end-to-end pipelines
- Edge case / destructive / boundary-condition tests
- Extended tests targeting surviving mutants and coverage gaps
- mutate.yml and dashboard.yml are now portable to any CPAN module.
Copying both files to another module's .github/workflows/ directory
will produce a working mutation test run and coverage dashboard
without modification. ATG-specific behaviour is gated on
github.repository = nigelhorne/App-Test-Generator.
- dashboard.yml: added missing local/bin PATH setup so installed
ATG scripts are found correctly on non-ATG modules.
- mutate.yml: Sample/Module.pm excluded from mutation scoring to
remove 30 artificial survivors from the overall score.
- Emitter::Perl mutation score improved from 37.5% to 100% through
targeted direct assertions on all _emit_*() helper return values.
- t/test-generator-index.t: 33 new tests for detect_perl_version_cliff(),
parse_version(), extract_perl_versions(), make_key(),
confidence_score(), and perldelta_url().
0.33 Tue Apr 21 16:09:24 EDT 2026
- Fixed NumericBoundary mutator incorrectly identifying the '<' in
open() mode strings and readline operators (<$fh>) as numeric
comparison operators, producing invalid mutants that caused
compilation errors in the mutated code. Two guards added to
mutate(): skip any operator whose next sibling is a PPI symbol
token (catching <$fh>), and skip operators not parented by a
PPI::Statement, PPI::Structure::Condition, or
PPI::Structure::Block (catching quoted mode strings in open()
argument lists). The same readline guard is also applied inside
the transform closure to prevent mutation at apply time if PPI
tokenisation differs between the analysis and mutation documents.
- Rationalised program names so that they can be used in other modules
without the need to have copies
- Added detect_prereq_version_cliffs() to cross-reference declared
prerequisites from the MetaCPAN API against installed module
versions reported by CPAN Testers. Flags cases where failing
reports have an installed version below the declared minimum,
surfacing dependency version mismatches (e.g. Test::Mockingbird
0.02 installed where >= 0.03 is required) that the existing
detect_version_cliffs() misses because it only examines prereq
metadata within the reports themselves. Falls back to scanning
raw CPAN Testers report output for version complaint patterns if
the MetaCPAN API is unavailable. Results are merged into the
existing Dependency Version Cliffs section alongside findings
from detect_version_cliffs(). Added report HTML caching in
fetch_report_html() to avoid fetching the same report twice when
both aggregate_dependency_stats() and detect_prereq_version_cliffs()
process the same GUIDs.
- bin/fuzz-harness-generator: fixed shell injection risk in final
system() call, replaced string eval with block eval for JSON
module loading, added explicit use File::Spec, normalised if()
spacing throughout, added structured comment blocks for all four
helper functions, added YAML input validation before calling
generate()
- bin/generate-test-dashboard: removed duplicate cover -report json
step that was running twice per dashboard build
- bin/test-generator-index: added structured comment blocks for six
private analysis functions; fixed detect_universal_failure() to
partition genuine FAILs from NAs before counting, returning undef
when all reports are NA so detect_perl_version_cliff() can
diagnose correctly
- SchemaExtractor.pm: added structured comment blocks for all private
methods (Pass 3)
- App::Test::Generator: added relationships_code emission so schemas
containing relationship metadata produced by SchemaExtractor are
serialised into the generated test file
- Template test.tt: added semantic type test cases for filepath
(including Windows paths, reserved device names, UNC paths),
email, date_string, and iso8601_string; added relationship test
loop covering all six relationship types (mutually_exclusive,
required_group, conditional_requirement, dependency,
value_constraint, value_conditional); added float edge cases for
Inf, -Inf, and NaN with correct status handling based on whether
max is defined
0.32 Sun Apr 12 08:16:47 EDT 2026
- Added detect_scattered_failures() root cause detector. Surfaces a
weak-confidence advisory when failures and passes coexist across
2 or more common Perl series with no detectable version cliff or
OS pattern, suggesting flaky tests, optional dependency
differences, or CGI environment assumptions rather than a
compatibility issue. Complements detect_universal_failure() which
handles the opposite case of near-total failure. Confidence is
intentionally set to 0.40 (Weak) since this is a catch-all
signal rather than a precise diagnosis, ensuring it appears below
stronger signals in the root causes table when multiple detectors
fire simultaneously.
- Added detect_universal_failure() root cause detector. Surfaces a
high-confidence warning when failures occur across 3 or more
distinct Perl versions and 2 or more OS types with fewer than
10% passing reports, indicating a likely broken release rather
than a version- or platform-specific compatibility issue. Likely
causes listed in evidence: missing file in tarball, broken
Makefile.PL, or undeclared dependency. Integrated into
detect_root_causes() where it is evaluated first and sorted by
confidence alongside the existing OS, locale, and Perl version
cliff detectors.
- Fixed blib/ paths appearing in coverage table instead of lib/
paths. Devel::Cover instruments blib/ during testing; paths are
now normalised to lib/ for display, with deduplication against
any native lib/ entry.
- Fixed structural coverage percentages in Executive Summary and
Structural Coverage sections showing ~24% instead of ~93%.
_coverage_totals now aggregates from individual own-project files
rather than Devel::Cover's pre-aggregated Total key which
includes all instrumented CPAN dependencies.
- Fixed cyclomatic complexity badge colour and tooltip inverted.
High complexity now correctly shows red/Needs improvement;
low complexity shows green/Good. Second condition also fixed
to use $complexity rather than $score.
0.31 Fri Apr 10 08:07:40 EDT 2026
- Added TER3 (LCSAJ path coverage) column to mutation files table in
the test dashboard index, showing percentage and raw fraction
(e.g. "71.8% (352/491)")
- Added TER1, TER2 and TER3 metrics to per-file mutation report pages,
replacing the plain "Statement" and "Branch" labels with their
formal Test Effectiveness Ratio names
- Renamed the LCSAJ column header in the mutation files table to TER3
- With min set to zero on an integer, sometimes negative or floats could be created - added abs() and int() calls as needed
- Added --generate_mutant_tests=DIR option to generate_index.pl to
produce a timestamped test stub file (t/mutant_YYYYMMDD_HHMMSS.t)
for surviving mutants. High/Medium difficulty survivors get TODO
test stubs; Low difficulty survivors get comment-only hints.
Multiple mutations on the same source line are deduplicated into
one stub listing all variants — one good test kills them all.
Boundary value suggestions are generated for numeric mutations,
clamped and deduplicated for non-negative contexts such as
scalar() and length(). Environment variable hints are added where
the source line references $ENV{...}. The enclosing subroutine
name is shown in each stub for navigation context. File is skipped
if there are no survivors or low-difficulty hints to report.
- LCSAJ path dots on per-file mutation pages are now coloured blue
(covered) or red (not covered), based on whether any line in the
path range was executed during testing. Uncovered paths show
[NOT COVERED] in the hover tooltip. The LCSAJ legend is updated
to explain both colours.
- Replaced TER3-only column in mutation files table with a
TER1 / TER2 / TER3 triple, each component shown as a
colour-coded badge (green/yellow/red). TER1=Statement,
TER2=Branch, TER3=LCSAJ path coverage. Any component
without data shows a grey n/a badge. Column header carries
a tooltip defining all three metrics.
- Added --generate_test=mutant option to generate_index.pl (used
alongside --generate_mutant_tests=DIR). For NUM_BOUNDARY survivors,
attempts to produce a runnable YAML schema file in t/conf/ using
App::Test::Generator::SchemaExtractor rather than a TODO stub.
The schema is augmented with boundary values from the surviving
mutant (the exact boundary value plus one either side) and picked
up automatically by t/fuzz.t on the next test run. Falls back to
a TODO stub if SchemaExtractor fails, the enclosing sub cannot be
found, or confidence in the extracted schema is too low. The
--generate_test option is designed to accept future classes beyond
'mutant'. Updated generate_test_dashboard Step 7 to pass
--generate_test=mutant by default.
- Made output_dir optional in App::Test::Generator::SchemaExtractor
new() -- it is now only required if schema files will be written.
extract_all() gains a no_write option to suppress file output and
return schemas only, for use by callers that want to inspect or
augment schemas before deciding where to write them.
- Added --generate_fuzz flag to generate_index.pl. Scans t/conf/
for existing YAML schema files and writes timestamped augmented
copies (mutant_fuzz_YYYYMMDD_HHMMSS_FUNCTION.yml) with boundary
values from surviving NUM_BOUNDARY mutants merged in. The original
schema is never modified. Augmented schemas are picked up
automatically by t/fuzz.t. Schemas with no matching survivors are
skipped (with a verbose note). Boundary values are merged into
whichever edge key already exists in the schema (edge_case_array
or edge_cases), with deduplication. Schemas already prefixed with
mutant_fuzz_ are skipped to prevent cascading augmentation.
Updated generate_test_dashboard Step 7 to pass --generate_fuzz.
0.30 Thu Apr 2 07:17:16 EDT 2026
Added mutation levels.
Setting to fast will reduce the number of mutants by deduping and removing unnecessary mutants.
See App::Test::Generator::Mutator::_is_redundant_level for a list of those optimised out
Added basic LSCAJ data to the test dashboard.
Added simple security string testing.
Don't push too hard on builtins as they don't have good parameter validation.
Ensure adding only the byte order marker honours $min
More use of _DESCRIPTION
0.29 Thu Feb 26 12:57:59 EST 2026
Added mutation testing
Added guided testing to extract-schemas
Some routines were incorrectly labelled as getter routines
Added more edge cases
Added fallback to extract parameters from classic Perl body styles
Added Type::Param support (https://github.com/nigelhorne/App-Test-Generator/issues/4)
Getter routines take no arguments
Fixed string testing when both min and max are given
Don't give $class or $self as parameters
No input/output no longer croaks, because there are now a few tests that can be run
Add a basic hashref to mandatory args
Use UUID::Tiny and Readonly::Values::Boolean
0.28 Mon Feb 9 19:54:59 EST 2026
Latest test dashboard
Some getter/setter routines were missed
Test getset routines
Added 'isa' test. Tests code dies when passed the wrong type of object
0.27 Thu Jan 29 07:54:01 EST 2026
Fewer false positives for "optional"
Sometimes "new:" was missing
If a return value is 'length($foo)' it's an integer with a minimum value of 0
Some random strings were mistakenly sent as tests that were not in the memberof set
0.26 Mon Jan 12 07:57:54 EST 2026
Added strict_pod mode to SchemaExtractor, which croaks if it finds discrepancies between the POD and code
Improved detection of optional/required arguments
Improved detection of integer output types
Improved POD parser of parameters
Added --dry-run
Improved config validation
Better call of object methods in transforms
Test the sanity of Unicode outputs
Idempotent tests
Prevent silent duplicate method overwrites
Implement confidence_threshold
Refactor _detect_accessor_methods
Only generate one of call_code and position_code
Check for global side effects
Sanity test on DIES
Added timeout tests to ensure the test doesn't hang
0.25 Thu Jan 1 15:41:06 EST 2026
Ternary ? 1 : 0 return is taken to be an indicator of boolean returns
Now parses this pod to know that it's supposed to return a string:
Returns the sanitized HTML string
Use :: to call methods in non-oo modules, rather than ->
Flag when type is set to object by can is not set
0.24 Sun Dec 28 15:10:09 EST 2025
Return chances of false positive file path semantics
If the type of an variable can't be determined, guess at string, but lower the confidence level
Error Message Extraction:
Capture error messages from die/croak/confess
Use them to understand what makes input invalid die "Age must be positive" if $age <= 0;
Store that a negative/zero age is invalid
Use test of scalar(@_) to check for number of optional/mandatory args
Throw a float at a routine that only takes integers - it should error
Added example extraction
Parse POD for example calls and extract valid parameter combinations =head2 SYNOPSIS my $obj = Module->new(foo => 'bar', count => 5);
Store these as known-good test values
Corpus bool testing now uses ok rather than is
Ensure new doesn't refer to other packages
Set the isa property in the new function
0.23 Wed Dec 24 10:48:03 EST 2025
Fix https://www.cpantesters.org/cpan/report/079dba46-d92f-11f0-9642-dde56d8775ea
Don't give false negative that memberof isn't known
SchemaExtractor output files now end in .yml not .yaml
Ensure rand_str isn't called when memberof is set
When test_empty is set, do the right thing for memberof
0.22 Sun Dec 21 20:59:58 EST 2025
Added better API to generate
Added parameter relationship detection (mutually exclusive, required groups, conditional requirements, dependencies, value constraints)
Improved Object Detection
Detect factory methods that return instances
Recognize singleton patterns
Identify when constructor needs specific parameters
Handle inheritance (when parent class new() is needed)
Better Default Value Extraction
$param = $param || 'default_value'
$param //= 'default_value'
$param = defined $param ? $param : 'default'
Expanded _detect_list_context to
Better distinguish scalar vs list returns
Detect void context methods (those that modify state)
Recognize chaining patterns more reliably Identify error return conventions (undef on failure, etc.)
Add confidence scoring transparency, explain WHY confidence is what it is:
_analysis: confidence_factors:
- "POD documentation present (+30)"
- "Type validation in code (+20)"
- "No constraint information (-10)"
Add support for:
- Signatures (sub foo($x, $y = 5) { })
- Postfix dereferencing ($array->@*)
- Subroutine attributes (sub foo :Returns(Int) { })
- Field declarations (Perl 5.38+)
Added a new section to generated YAML:
yamltest_hints: boundary_values: [0, 1, 100, 255]
detected from code invalid_inputs: ['', undef, -1]
from validation checks equivalence_classes: []
0.21 Sun Dec 14 08:07:09 EST 2025
Schemaextractor: don't put the package name as the argument
Validate config settings better
Fix max string testing with non-ASCII characters
Changed rand_str to be a unified generator that randomly produces codepoint strings,
grapheme clusters, ZWJ emoji sequences, or aggressive Unicode fuzz strings
Schemaextractor: Added advanced type detection for DateTime objects, file handles, coderefs, and enum validation patterns
Added enum as a synonym of memberof
Added tests for unix_timestamp semantic type
0.20 Fri Dec 5 07:53:43 EST 2025
Added the --version flag to fuzz-harness-generator
Ensure the max value of string is honoured better
Fix array context detection to only match return statements
Improve chances of detecting a boolean output
Make the list context detection more specific
0.19 Wed Nov 26 07:50:22 EST 2025
Fixed the loop iterating over mandatory args
Ensure mandatory_args honours matches
0.18 Tue Nov 25 11:58:12 EST 2025
Removed one place sending '' if test_empty was disabled
Don't send long strings if matches is set
Fixed handling of position code with more than argument
Renamed the sample module
0.17 Tue Nov 25 10:24:36 EST 2025
Improved TEST_VERBOSE output for Corpus tests
Make the strings more random
Added test_non_ascii (default 1) to the configuration matrix
Removed all legacy conf code
Re-ordered the POD
Don't give 42 if max < 42
Added App::Test::Generator::SchemaExtractor
0.16 Thu Nov 20 08:38:17 EST 2025
Use Data::Random::String to generate the string
Use Data::Random::Structure to generate references to hashes and arrays
Remove legacy support for Perl config files
Handle the case when all inputs should cause a die(),
allows App::Test::Generator to sanity test itself
If something is expected to die, don't look at its return code
More boolean edge cases
Don't pass _NAME to validate_strict
When a routine dies, it shouldn't return anything
Begin to use semantic types
Use Test::LectroTest to create more tests
Added custom properties
Separated the Template into its own class to ease maintenance
Use reusable functions to generate cases
Rand_set returns and array, so look at the first element
0.15 Tue Nov 11 19:30:18 EST 2025
Documented how to Fuzz test a CPAN module
populate_positions - don't attempt to deref a scalar
carp when undef is a corpus test case, but test_undefs is not set
Improved handling of cases array for WARNS and DIES
Improved handling of validating position settings in config files
Croak if the schema_file can't be opened
Close stdin and use /dev/null
Fixed logic in testing short strings when min >= 2
0.14 Fri Nov 7 20:26:32 EST 2025
Fixed logic when neither memberof nor notmemberof were defined,
which could cause an empty notmemberof array to be created
Transform: better number of items for arrayrefs
At least one of module and function must be defined
Don't send wrong data types in transform testing built ins
Document the edge_cases_array and ensure all tests in that array are run
Better handing of "input: undef" in the config file
Fix handling of matches in output schemas
0.13 Tue Nov 4 11:52:49 EST 2025
Added better documentation for transforms
transform: filled in more types for the foundation set of data
Added "value" to the transform keywords
Refactored and enabled the code to validate that $module exists and is installed
Added deprecation notice about loading configs from perl code
Allow yes/no as booleans in schema config settings
Fix sprintf issue when the text contains a % sign
0.12 Sun Nov 2 19:38:10 EST 2025
undef can now be used to indicate no input or output
Sending with no args if now configured using "test_undef" rather than "test_empty"
Use Data::Random::String::Matches to create random string that matches a regex
Flag keywords that are yet to be handled
$module can now be set to "builtin"
Initial basic support for positional arguments and transforms
0.11 Sun Oct 26 14:39:19 EDT 2025
Only add to candidate_bad those test cases which are requested
Better matches and nomatch tests
Only load JSON module when dedup is enabled
Added the -o and -r options to bin/fuzz-harness-generator
0.10 Wed Oct 22 09:23:38 EDT 2025
Send wrong data types - should die, e.g. string to an integer field
Fix syntax error when Math::Simple is installed
Added test_empty config (default is on)
0.09 Sun Oct 19 21:14:29 EDT 2025
Better handling of false/true in config files
Better handling of regex into the input structure
0.08 Thu Oct 16 20:44:22 EDT 2025
Allow $new to be set but not defined, this will call new() with no arguments
Started to add support for "nomatch"
Allow specs that specify no input if they have output
If a routine takes no input, it won't die if it has no input
0.07 Tue Oct 14 16:36:33 EDT 2025
Bump minimum versions
Added float that was missing some places
Use Data::Random
Fix GitHub#3
Show how to use this module to automatically schedule random tests
Test for code doing if($string) rather than if(defined($string)) which is confused if $string is '0'
Allow the special word "undef" in the YAML specification for output
0.06 Fri Oct 10 08:55:36 EDT 2025
Allow real config files to be read
Validate the configuration file
0.05 Thu Oct 9 18:23:03 EDT 2025
Removed duplicate regex candidate table
Validate that the corpus inputs are arrayrefs
Only load Class::Simple when needed
Use rel2abs
Load the configuration file in a (slightly) safer way
The test code is now in a template toolkit
Improved random test generator, more knowledgable about min/max, memberof and matches
0.04 Wed Oct 8 08:39:49 EDT 2025
Add fuzzy regex generator
Do basic hard-coded tests where possible, to get it started
Allow pathnames in the module name
Added qwrap - GitHub#1 - thanks to neo1ite
Don't try to fuzz input if no %input is given
Added %config - GitHub#2
Generate tests for routines that take one unnamed parameter - GitHub#2
Added fallback for perl_quote for hashes and objects
0.03 Mon Sep 29 18:18:36 EDT 2025
If minimum is not set, verify 0 or empty fields are allowable
Added the testing dashboard
Fixed handling of memberof in input/output array creation
If TEST_VERBOSE is set, print the generated dataset when running it
Always ensure mandatory strings are passed when testing other arguments
rand_int and rand_numb now also sometimes return very large and very small numbers
Put utf-8 and NUL bytes into strings
0.02 Sun Sep 28 09:03:49 EDT 2025
Use gtar on OS/X to generate the distro
Added edge case test generator for booleans and memberof
0.01 Sun Sep 28 08:43:35 EDT 2025
First draft