NAME
app-test-generator-mutate - Run mutation testing against a Perl test suite
SYNOPSIS
app-test-generator-mutate [options]
app-test-generator-mutate --lib lib --tests t
app-test-generator-mutate --file lib/My/Module.pm
app-test-generator-mutate --json mutation.json
app-test-generator-mutate --min-score 75QUICK START
app-test-generator-mutate --lib lib --min-score 85 --json mutation.json --html mutation_html
open mutation_html/index.htmlNumeric Boundary Mutants
Kill Numeric Boundary Mutants first, these are the easiest wins. For example, NUM_BOUNDARY_1295 means something like if ($x 10)> became if ($x = 10)> or if ($x 9)>. If that survived, it means, there is a missing edge value. Numeric mutations are important because they reveal missing edge coverage. This example means line 1295.
So if that line contains something like this:
if(((scalar keys %input) == 1) && exists($input{'type'}) && !ref($input{'type'})) {You need to add a test where
%input contains more than one key
One of them is type
And behavior must be different
For example, if you have a test with
%input = ( type => 'string' )add a test which sets
%input = (
type => 'string',
something_else => 'value'
)Conditional Inversions
Then kill Conditional Inversions, for example, COND_INV_1186, where unless (-f $file) became if (-f $file). If that survives, test did not assert the negative case.
Focus by file, if one file contributes 200 survivors, that's the weakest module.
Frequently re-run. The loop should be: add 5-10 targeted tests, re-run mutation tool, watch score climb, repeat.
DESCRIPTION
This command-line tool performs mutation testing on a Perl codebase.
It scans one or more .pm files, generates code mutations using App::Test::Generator::Mutator, and runs the project's test suite against each mutated version inside an isolated workspace.
For each generated mutant:
The mutant is applied in a temporary workspace.
The mutated file is syntax-checked.
The test suite is executed using
prove.If the tests fail, the mutant is considered killed.
If the tests pass, the mutant is considered survived.
A mutation score is then calculated:
(killed / total) * 100Mutation testing measures the effectiveness of a test suite. A higher mutation score indicates that the tests are better at detecting behavioral changes in the code.
OPTIONS
--lib <dir>
Directory containing Perl modules to mutate.
Defaults to lib.
--file <file>
Mutate a single file instead of scanning the entire --lib directory.
--tests <dir>
Directory containing test files.
Defaults to t.
--min-score <int>
Minimum acceptable mutation score (percentage).
If the final score is below this value, the program exits with a non-zero status.
--json <file>
Write mutation results to the specified JSON file.
The output structure:
{
score => "85.32",
total => 120,
killed => 102,
survived => [ ... mutant IDs ... ]
}--fail-fast
(Reserved for future use.)
--timeout <seconds>
(Reserved for future use.)
--verbose
Print progress information.
--quiet
Suppress final summary output.
EXIT CODES
- = 0
-
Success and mutation score meets minimum threshold.
- = 1
-
Mutation score below
--min-score. - = 2
-
Baseline test suite failed before mutation testing began.
- = 3
-
Invalid command-line options.
WORKFLOW
The tool performs the following steps:
Collect target files (either a single file or all
.pmfiles under--lib).Run baseline tests to ensure the suite passes before mutation.
Generate mutants for each file.
Apply each mutant in isolation and re-run the test suite.
Calculate and report mutation statistics.
WORKFLOW DIAGRAM
The mutation testing process follows this execution flow:
┌───────────────────────────────┐
│ Start │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Collect Target Files │
│ --file OR scan --lib/*.pm │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Run Baseline Tests │
│ prove -l t │
└───────────────┬───────────────┘
│
Baseline OK? ── No ──► Exit (code 2)
│
Yes
│
▼
┌───────────────────────────────┐
│ For Each File │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Generate Mutants │
│ (conditional flips, etc.) │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ For Each Mutant │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Prepare Workspace │
│ (isolated temp directory) │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Apply Mutant │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Syntax Check │
│ perl -c mutated_file.pm │
└───────────────┬───────────────┘
│
Compiles? ── No ──► Skip Mutant
│
Yes
│
▼
┌───────────────────────────────┐
│ Run Test Suite │
│ prove t │
└───────────────┬───────────────┘
│
Tests Fail? ── Yes ──► Killed++
│
No
│
▼
Survived++
│
▼
┌───────────────────────────────┐
│ Repeat for Next Mutant │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Calculate Mutation Score │
│ (killed / total) * 100 │
└───────────────┬───────────────┘
│
▼
┌───────────────────────────────┐
│ Print Report / Write JSON │
└───────────────┬───────────────┘
│
▼
FinishAUTHOR
Nigel Horne