NAME
test-generator-mutate - Run mutation testing against a Perl test suite
SYNOPSIS
test-generator-mutate [options]
test-generator-mutate --lib lib --tests t
test-generator-mutate --file lib/My/Module.pm
test-generator-mutate --json mutation.json
test-generator-mutate --min-score 75QUICK START
test-generator-mutate --lib lib --min-score 85 --json mutation.jsonSkipping Lines from Mutation
Some routines involve logic that must not be mutated because doing so would cause the test suite to hang rather than produce a clean pass/fail result. Two common cases:
- Recursion base cases
-
The
ConditionalInversionandBooleanNegationoperators will negate anyifcondition, including the guard that terminates a recursive function. A mutant that removes the base case causes infinite recursion and the per-mutantproverun hangs indefinitely (there is currently no per-mutant timeout). Protect the base-case condition:sub factorial { my ($n) = @_; ## MUTANT_SKIP_BEGIN return 1 if $n <= 1; ## MUTANT_SKIP_END return $n * factorial($n - 1); } - Process-management side-effects
-
Forking, signalling, and
waitpidare inherently difficult to assert against in a unit test. A mutant that corrupts a PID or signal value may cause the test suite to hang or emit warnings from anENDblock:sub disable_hot_reload { ## MUTANT_SKIP_BEGIN kill 'HUP', $pid if $pid; waitpid $pid, 0; ## MUTANT_SKIP_END }
To exclude a block of lines from mutation testing entirely, wrap it with ## MUTANT_SKIP_BEGIN and ## MUTANT_SKIP_END annotations as shown above.
Skipped lines are silently excluded from the mutation candidate list and do not appear in the killed or survived counts. The number of skipped lines is shown in the per-file mutation report.
Mismatched markers (a ## MUTANT_SKIP_BEGIN with no matching ## MUTANT_SKIP_END, or vice versa) are fatal errors.
Numeric Boundary Mutants
Kill Numeric Boundary Mutants first, these are the easiest wins. For example, NUM_BOUNDARY_1295 means something like if ($x 10)> became if ($x = 10)> or if ($x 9)>. If that survived, it means, there is a missing edge value. Numeric mutations are important because they reveal missing edge coverage. This example means line 1295.
So if that line contains something like this:
if(((scalar keys %input) == 1) && exists($input{'type'}) && !ref($input{'type'})) {You need to add a test where
%input contains more than one key
One of them is type
And behavior must be different
For example, if you have a test with
%input = ( type => 'string' )add a test which sets
%input = (
type => 'string',
something_else => 'value'
)Conditional Inversions
Then kill Conditional Inversions, for example, COND_INV_1186, where unless (-f $file) became if (-f $file). If that survives, test did not assert the negative case.
Focus by file, if one file contributes 200 survivors, that's the weakest module.
Frequently re-run. The loop should be: add 5-10 targeted tests, re-run mutation tool, watch score climb, repeat.
DESCRIPTION
This command-line tool performs mutation testing on a Perl codebase.
It scans one or more .pm files, generates code mutations using App::Test::Generator::Mutator, and runs the project's test suite against each mutated version inside an isolated workspace.
For each generated mutant:
The mutant is applied in a temporary workspace.
The mutated file is syntax-checked.
The test suite is executed using
prove.If the tests fail, the mutant is considered killed.
If the tests pass, the mutant is considered survived.
A mutation score is then calculated:
(killed / total) * 100Mutation testing measures the effectiveness of a test suite. A higher mutation score indicates that the tests are better at detecting behavioral changes in the code.
OPTIONS
--lib <dir>
Directory containing Perl modules to mutate.
Defaults to lib.
--file <file>
Mutate a single file instead of scanning the entire --lib directory.
--tests <dir>
Directory containing test files.
Defaults to t.
--changed_only
Only mutate files that were changed in the most recent commit, as determined by git diff --name-only HEAD~1 HEAD. Files not changed in the current commit retain their mutation results from the previous dashboard run. This significantly reduces CI runtime while preserving accuracy for the files that actually changed.
--exclude <path>
Exclude files matching the given path fragment from mutation testing. May be specified multiple times. For example:
--exclude lib/Devel --exclude lib/App/Test/Generator/Sample--base_sha <sha>
The git commit SHA to use as the base when computing which files have changed under --changed_only. Defaults to HEAD~1.
Use this when automated commits (such as coverage snapshots or generated test stubs) have landed on top of your last code commit, causing HEAD~1 to point to an automated commit rather than a real change. Typically set by the CI workflow using the output of git log filtered to exclude automated commits.
--min-score <int>
Minimum acceptable mutation score (percentage).
If the final score is below this value, the program exits with a non-zero status.
--json <file>
Write mutation results to the specified JSON file.
The output structure:
{
score => "85.32",
total => 120,
killed => 102,
survived => [ ... mutant IDs ... ]
}--cover_json <file>
The location of the file generated by cover -report json. That file is used to generate an approximation for an LCSAJ table.
--fail-fast
(Reserved for future use.)
--mutation_level <full|fast>
Setting to fast removes redundant mutations and dedups mutations before running. The default is full.
--timeout <seconds>
(Reserved for future use.)
--verbose
Print progress information.
--quiet
Suppress final summary output.
EXIT CODES
- = 0
-
Success and mutation score meets minimum threshold.
- = 1
-
Mutation score below
--min-score. - = 2
-
Baseline test suite failed before mutation testing began.
- = 3
-
Invalid command-line options.
WORKFLOW
The tool performs the following steps:
Collect target files (either a single file or all
.pmfiles under--lib).Run baseline tests to ensure the suite passes before mutation.
Generate mutants for each file.
Apply each mutant in isolation and re-run the test suite.
Calculate and report mutation statistics.
WORKFLOW DIAGRAM
The mutation testing process follows this execution flow:
┌───────────────────────────────┐
│ Start │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Collect Target Files │
│ --file OR scan --lib/*.pm │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Run Baseline Tests │
│ prove -l t │
└───────────────┬───────────────┘
│
Baseline OK? ── No ──► Exit (code 2)
│
Yes
│
▼
┌───────────────────────────────┐
│ For Each File │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Generate Mutants │
│ (conditional flips, etc.) │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ For Each Mutant │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Prepare Workspace │
│ (isolated temp directory) │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Apply Mutant │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Syntax Check │
│ perl -c mutated_file.pm │
└───────────────┬───────────────┘
│
Compiles? ── No ──► Skip Mutant
│
Yes
│
▼
┌───────────────────────────────┐
│ Run Test Suite │
│ prove t │
└───────────────┬───────────────┘
│
Tests Fail? ── Yes ──► Killed++
│
No
│
▼
Survived++
│
▼
┌───────────────────────────────┐
│ Repeat for Next Mutant │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Calculate Mutation Score │
│ (killed / total) * 100 │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Print Report / Write JSON │
└───────────────┬───────────────┘
│
▼
FinishAUTHOR
Nigel Horne