NAME
App::Test::Generator::Mutator - Generate and apply mutation tests
VERSION
Version 0.49
SYNOPSIS
A "mutant" is a copy of your code with one small, deliberate mistake. If your tests still pass, the mutant "survived" and your tests missed the mistake. If a test fails, the mutant was "killed", which is good.
Pattern 1: list the mutants for a file. Nothing is changed on disk.
use App::Test::Generator::Mutator;
my $mutator = App::Test::Generator::Mutator->new(
file => 'lib/My/Module.pm',
lib_dir => 'lib',
);
my $mutants = $mutator->generate_mutants();
printf "Generated %d mutants\n", scalar @{$mutants};
printf "line %d: %s\n", $_->line, $_->description for @{$mutants};
Pattern 2: test every mutant. Run this from the top directory of the project. The mutated file is copied over the real file for the test run and the real file is always put back. This is what bin/test-generator-mutate does.
my $file = 'lib/My/Module.pm';
my $read = sub { open my $fh, '<', $_[0]; local $/; <$fh> };
my $original = $read->($file);
for my $m (@{$mutants}) {
my $workspace = $mutator->prepare_workspace(); # clean copy each time
$mutator->apply_mutant($m);
my $mutated = $read->("$workspace/$file");
{ open my $fh, '>', $file; print $fh $mutated }
my $survived = $mutator->run_tests();
{ open my $fh, '>', $file; print $fh $original }
printf "%s: %s\n", ($survived ? 'SURVIVED' : 'KILLED'), $m->description;
}
Pattern 3: use fast mode to get fewer mutants and a shorter run.
my $quick = App::Test::Generator::Mutator->new(
file => 'lib/My/Module.pm',
mutation_level => 'fast',
);
my $fewer = $quick->generate_mutants();
printf "fast: %d mutants, full: %d mutants\n",
scalar @{$fewer}, scalar @{$mutants};
Pattern 4: protect lines of your own code from mutation. Put these two comment lines around them in the file to be mutated. Each marker must be alone on its line.
## MUTANT_SKIP_BEGIN
return 1 if $n <= 1; # no mutant will be made for this line
## MUTANT_SKIP_END
DESCRIPTION
App::Test::Generator::Mutator is a mutation engine that programmatically alters Perl source files to evaluate the effectiveness of a project's test suite. It analyses modules, generates systematic code mutations (such as conditional inversions, logical operator changes, and numeric boundary flips), and applies them within an isolated workspace so tests can be executed safely against each modified variant.
By tracking which mutants are killed (cause tests to fail) versus those that survive (tests still pass), the module enables calculation of a mutation score, providing a quantitative measure of how well the test suite detects unintended behavioural changes.
new
Construct a new Mutator for a given source file.
my $mutator = App::Test::Generator::Mutator->new(
file => 'lib/My/Module.pm',
lib_dir => 'lib',
mutation_level => 'full',
);
Arguments
filePath to the Perl source file to mutate. Required. Must exist on disk.
lib_dirRoot library directory. Optional - defaults to
lib. Use a relative path.filemust begin with this path.mutation_levelControls the breadth of mutation.
fullapplies all mutations;fastdeduplicates and removes redundant mutants first. Optional - defaults tofull.
Returns
A blessed hashref. Croaks if file is missing or does not exist. The file is only checked here. It is read later, by generate_mutants.
API SPECIFICATION
input
{
file => { type => SCALAR },
lib_dir => { type => SCALAR, optional => 1 },
mutation_level => { type => SCALAR, optional => 1 },
}
output
{
type => OBJECT,
isa => 'App::Test::Generator::Mutator',
}
EXAMPLE
my $m = App::Test::Generator::Mutator->new(
file => 'lib/Acme/Widget.pm',
mutation_level => 'fast',
);
MESSAGES
file required-
filewas not supplied. file not found: PATH-
filewas supplied but does not exist on disk.
The mathematical description is in "FORMAL SPECIFICATION".
generate_mutants
Parse the target file and generate all mutants by running each registered mutation strategy against the PPI document.
my $mutants = $mutator->generate_mutants(); # scalar context: arrayref
for my $m (@{$mutants}) { ... }
my @mutants = $mutator->generate_mutants(); # list context: flat list (backward-compat)
Arguments
None beyond $self.
Returns
An arrayref of App::Test::Generator::Mutant objects. In fast mode, redundant and duplicate mutants are removed before returning. Lines within ## MUTANT_SKIP_BEGIN / ## MUTANT_SKIP_END annotation blocks are excluded from the candidate list entirely. After this method returns, $self->{skip_lines} contains a hashref mapping excluded line numbers to 1.
API SPECIFICATION
input
{
self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
}
output
{
type => ARRAYREF,
elements => { type => OBJECT, isa => 'App::Test::Generator::Mutant' },
}
EXAMPLE
my $mutants = $mutator->generate_mutants();
printf "%d mutants generated\n", scalar @{$mutants};
for my $m (@{$mutants}) {
printf "line %d: %s\n", $m->line, $m->description;
}
MESSAGES
Unable to parse FILE-
PPI could not parse the source file (syntax error or unreadable file). FILE is the path passed to
new. FILE: MUTANT_SKIP_BEGIN at line N with no prior MUTANT_SKIP_END-
A
## MUTANT_SKIP_BEGINmarker was found while already inside a skip block. FILE: MUTANT_SKIP_END at line N with no matching MUTANT_SKIP_BEGIN-
A
## MUTANT_SKIP_ENDmarker was found with no preceding## MUTANT_SKIP_BEGIN. FILE: MUTANT_SKIP_BEGIN at line N has no matching MUTANT_SKIP_END-
The source file ended while still inside a skip block.
The mathematical description is in "FORMAL SPECIFICATION".
PSEUDOCODE
parse file with PPI
scan lines for MUTANT_SKIP_BEGIN / MUTANT_SKIP_END pairs, giving skip_lines
for each registered mutation strategy:
skip if strategy does not apply_to(doc)
for each mutant from strategy->mutate(doc):
include unless mutant.line is in skip_lines
if mutation_level == 'fast':
deduplicate and remove redundant mutants
return arrayref (or flat list in list context)
prepare_workspace
Prepare an isolated temporary workspace for a single mutant.
The entire lib_dir tree is copied into the workspace so that all module dependencies resolve correctly when the test suite runs against the mutant. Only after this copy is complete is the single target file overwritten by apply_mutant.
my $workspace = $mutator->prepare_workspace();
$mutator->apply_mutant($mutant);
local $ENV{PERL5LIB} = "$workspace/lib";
my $survived = (system('prove', 't') == 0);
Arguments
None beyond $self.
Returns
A string containing the absolute path to the temporary directory created. The directory is removed automatically when the program ends (File::Temp's CLEANUP => 1 behaviour). It is not removed earlier.
Side effects
Creates a temporary directory. Recursively copies lib_dir into it. Sets $self->{_workspace}, $self->{_relative}, and $self->{_lib_basename}. Does not modify $self->{lib_dir}.
Notes
Call prepare_workspace again before each apply_mutant call. apply_mutant does not undo the previous mutant, so two mutants applied to the same workspace are both present in the file.
Each call makes a new directory and a new full copy of lib_dir. The object then points at the new one. Older workspaces stay on disk until the program ends, so a long run needs disk space for one copy per call.
API SPECIFICATION
input
{
self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
}
output
{
type => SCALAR,
}
EXAMPLE
my $workspace = $mutator->prepare_workspace();
# workspace is an absolute temp dir path
# original lib_dir value is unchanged
printf "original lib_dir still: %s\n", $mutator->{lib_dir};
MESSAGES
dircopy failed: $!-
The
lib_dirtree could not be copied into the temporary workspace directory, usually a permissions error.
The mathematical description is in "FORMAL SPECIFICATION".
apply_mutant
Apply a single mutant's transform to the target file in the workspace.
$mutator->apply_mutant($mutant);
Arguments
$mutantAn App::Test::Generator::Mutant object whose
transformclosure will be applied to the workspace copy of the target file.
Returns
Nothing useful. Do not test the return value. Modifies the workspace copy of the target file in place.
Side effects
Overwrites the target file in the workspace with the mutated version. The real project file is never touched by this method.
API SPECIFICATION
input
{
self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
mutant => { type => OBJECT, isa => 'App::Test::Generator::Mutant' },
}
output
{ type => UNDEF }
EXAMPLE
for my $m (@{$mutants}) {
my $workspace = $mutator->prepare_workspace();
$mutator->apply_mutant($m);
# "$workspace/lib/My/Module.pm" now holds exactly one mutation
}
MESSAGES
Workspace not prepared -- call prepare_workspace first-
apply_mutantwas called beforeprepare_workspace. Relative path not set -- call prepare_workspace first-
Internal: the relative-path field was not set by
prepare_workspace. Failed to parse TARGET-
PPI could not parse the workspace copy of the target file.
Can't call method "transform" on an undefined value-
A Perl error, not a message from this module.
$mutantwasundef.
The mathematical description is in "FORMAL SPECIFICATION".
run_tests
Run the project's test suite and return whether all tests passed.
my $survived = $mutator->run_tests();
Arguments
None beyond $self.
Returns
A true value (1) if all tests passed (mutant survived). A false value (the empty string, not 0) if any test failed or prove could not be started (mutant killed). Test it with if, not with == 0.
Side effects
Executes an external process running the test suite. The output of prove goes to the terminal.
Notes
Runs prove -l t in the current directory. This tests the lib directory of the project, not the workspace. To test a mutant, copy the mutated file from the workspace over the project file first, and put the original back afterwards. See Pattern 2 in the "SYNOPSIS".
The prove program is looked for next to the running perl first, then in Perl's own bin directory, and only then on PATH.
There is no time limit. A mutant that causes an endless loop makes this call wait for ever.
API SPECIFICATION
input
{
self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
}
output
{ type => SCALAR }
EXAMPLE
my $survived = $mutator->run_tests();
if($survived) {
print "mutant survived\n";
} else {
print "mutant killed\n";
}
The mathematical description is in "FORMAL SPECIFICATION".
COMMON PITFALLS
- Calling
apply_mutantbeforeprepare_workspace -
apply_mutantrequiresprepare_workspaceto have been called first. The workspace holds the isolated copy oflib/that receives mutations. - Passing an absolute path as
lib_dir -
lib_dirmust be a relative path (e.g.lib). An absolute path causesapply_mutantto construct a doubled directory under the workspace and then fail to find the target file. - Checking
$mutator->{workspace}after the refactor -
Internal workspace state is stored in
_workspace,_relative, and_lib_basename(note the underscore prefix). The originallib_dirvalue is never overwritten. Old code that reads$mutator->{workspace}or$mutator->{relative}(without underscores) will not find these keys. - Forgetting that
run_testsdrivesprovefrom$^X -
run_testsresolvesprovefrom the same Perl binary used to run the script. If you shell out toprovedirectly in your own integration code, make sure you are using the matchingprove. generate_mutantsin list vs scalar context-
generate_mutantsreturns a flat list in list context and an arrayref in scalar context. Assign to an arrayref (my $m = $mutator->generate_mutants()) to guarantee you always get a reference regardless of calling context. - Applying two mutants to one workspace
-
apply_mutantchanges the workspace file and leaves it changed. If you apply a second mutant without a newprepare_workspacecall, the file holds both mutations and your results are wrong. - Expecting
run_teststo test the workspace -
run_testsruns the tests against the project's own lib directory. If you only callapply_mutantand thenrun_tests, the tests load the original code, every mutant "survives", and the kill rate is 0%. - An unknown
mutation_levelis not an error -
Only the exact text
fastturns on fast mode. Any other value, for exampleFastorquick, is accepted silently and works likefull. - How
undefand other false values are handled -
file => undefmakesnewcroak withfile required. An empty string makes it croak withfile not found.lib_dirandmutation_leveluse their default when the value isundef, an empty string or0. You cannot have a library directory that is really named0.apply_mutant(undef)stops with a Perl error, after the workspace check.run_testsreturns an empty string for "failed". It never returnsundef.A mutant's
line,originalordescriptionmay beundef. Fast mode treats each missing value as an empty string when it looks for duplicates.
- Extra arguments to
neware ignored -
A wrong spelling such as
libdirorlevelis not reported. The default value is used and no warning is given. - Skip markers must be alone on their line
-
## MUTANT_SKIP_BEGINand## MUTANT_SKIP_ENDare only seen when nothing else except spaces is on the same line. Markers cannot be nested. The two marker lines themselves are also skipped.
ENCODING
The
fileandlib_dirargumentsPaths are passed to the operating system unchanged. ASCII paths always work. For non-ASCII directory or file names, pass the path as bytes in the encoding of your file system (normally UTF-8).
Source file contents
The file to mutate is read and written by PPI as raw bytes. UTF-8 text, other non-ASCII text and emojis inside strings, comments and POD are copied to the mutated file byte for byte. They are not damaged and they do not need to be decoded first.
Text returned by a mutant
$mutant->original,$mutant->descriptionand$mutant->line_contentcan contain pieces of your source code. These are byte strings. If the source is UTF-8 and you print them through a UTF-8 file handle, decode them first withEncode::decode('UTF-8', $text).Skip markers
The markers are ASCII. They are found correctly in a file of any ASCII-compatible encoding, including UTF-8.
Error messages
Messages contain the file path exactly as you gave it.
LIMITATIONS
Single strategy registry
The four built-in mutation strategies are hardcoded in
new. There is no plugin mechanism for registering additional strategies without subclassing. A future version should accept astrategiesarrayref argument.No parallelism
run_testsis synchronous. For large test suites or large mutant sets, wall-clock time scales linearly. The in-place mutation strategy also serialises all mutants behind a single file lock.PPI re-parse per apply_mutant call
apply_mutantre-parses the workspace copy of the target file for every mutant. For very large single-file modules the PPI parse time may dominate.apply_mutant does not restore on abnormal exit
If the process is killed between the write and restore in
bin/test-generator-mutate, the project file is left mutated. Agit restore lib/...recovers it.
SEE ALSO
bin/test-generator-mutate- Devel::Mutator
AUTHOR
Nigel Horne, <njh at nigelhorne.com>
FORMAL SPECIFICATION
This section describes each method in Z notation. It is for readers who want an exact, mathematical statement. You do not need it to use the module.
How to read it: a name ending in ? is an input, a name ending in ! is an output, and a name ending in ' is the value after the call. Δ before the state name means "this call may change the object". Ξ means "this call does not change the object".
LEVEL == { 'full', 'fast' }
┌─ Mutator ─────────────────────────────────────────
│ file, lib_dir : PATH
│ mutation_level : STRING
│ skip_lines : ℙ ℕ₁ ∪ { undef }
│ workspace : PATH ∪ { undef }
│ relative : PATH ∪ { undef }
├───────────────────────────────────────────────────
│ workspace = undef ⇔ relative = undef
└───────────────────────────────────────────────────
new
┌─ New ─────────────────────────────────────────────
│ Mutator′
│ file? : VALUE
│ lib_dir?, mutation_level? : VALUE
│ result! : Mutator
├───────────────────────────────────────────────────
│ pre file? ≠ undef ∧ file? names an existing plain file
│ post file′ = file?
│ lib_dir′ = if lib_dir? is true
│ then lib_dir? else 'lib'
│ mutation_level′ = if mutation_level? is true
│ then mutation_level? else 'full'
│ skip_lines′ = undef ∧ workspace′ = undef ∧ relative′ = undef
│ result! = θ Mutator′
└───────────────────────────────────────────────────
file? = undef ⇒ croak 'file required'
file? is not an existing file ⇒ croak 'file not found: ' ⁀ file?
generate_mutants
┌─ GenerateMutants ─────────────────────────────────
│ Δ Mutator
│ result! : seq Mutant
│ all : seq Mutant
├───────────────────────────────────────────────────
│ pre file is valid Perl that PPI can parse
│ skip markers in file are paired and not nested
│ post skip_lines′ = { n : ℕ₁ | line n of file lies between
│ a BEGIN marker and its END marker,
│ both marker lines included }
│ all = every mutant that the four strategies
│ produce for file, in strategy order
│ ∀ m : ran result! • m ∈ ran all ∧ m.line ¬in; skip_lines′
│ mutation_level ≠ 'fast' ⇒
│ result! = all with skipped lines removed
│ mutation_level = 'fast' ⇒
│ (∀ i, j : dom result! • i ≠ j ⇒
│ key(result!(i)) ≠ key(result!(j)))
│ ∧ (∀ m : ran result! • ¬ redundant(m))
│ file′ = file ∧ lib_dir′ = lib_dir
│ workspace′ = workspace ∧ relative′ = relative
└───────────────────────────────────────────────────
key(m) == (m.line, m.original, m.description)
In list context the members of result! are returned as a list.
prepare_workspace
┌─ PrepareWorkspace ────────────────────────────────
│ Δ Mutator
│ result! : PATH
├───────────────────────────────────────────────────
│ pre lib_dir names a directory that can be copied
│ post result! names a directory created by this call
│ result! ≠ workspace
│ result!/basename(lib_dir) is a full copy of lib_dir
│ workspace′ = result!
│ relative′ = file with the leading lib_dir removed
│ file′ = file ∧ lib_dir′ = lib_dir
│ mutation_level′ = mutation_level
│ skip_lines′ = skip_lines
└───────────────────────────────────────────────────
the copy fails ⇒ croak 'dircopy failed: ' ⁀ reason
apply_mutant
┌─ ApplyMutant ─────────────────────────────────────
│ Ξ Mutator
│ mutant? : Mutant
│ target : PATH
├───────────────────────────────────────────────────
│ pre workspace ≠ undef ∧ relative is true
│ mutant?.transform is a CODE reference
│ target = workspace/basename(lib_dir)/relative
│ target is valid Perl that PPI can parse
│ post content′(target) = mutant?.transform(content(target))
│ content′(file) = content(file)
└───────────────────────────────────────────────────
workspace = undef ⇒ croak 'Workspace not prepared -- ...'
relative is false ⇒ croak 'Relative path not set -- ...'
target not parsed ⇒ croak 'Failed to parse ' ⁀ target
run_tests
┌─ RunTests ────────────────────────────────────────
│ Ξ Mutator
│ result! : { 1, '' }
├───────────────────────────────────────────────────
│ post result! = 1 ⇔ the command 'prove -l t', run in the
│ current directory, exits with status 0
└───────────────────────────────────────────────────
STATE DIAGRAM
The object has three states. The state says what has been prepared so far.
new(file => undef) or new(file => missing file)
+--------------------------------------------------+
| action: croak, no object |
v |
+-------------+ |
| NO OBJECT |--------------------------------------+
+-------------+
|
| new(file => existing file)
| action: store file, lib_dir, mutation_level;
| build the four mutation strategies
v
+---------------------+ generate_mutants()
| |-----------------------------------------+
| CONSTRUCTED | action: read file, set skip_lines, |
| (no workspace yet) | return the mutants (state stays same) |
| |<----------------------------------------+
| |
| | run_tests()
| |-----------------------------------------+
| | action: run 'prove -l t', return 1/'' |
| |<----------------------------------------+
| |
| | apply_mutant(m)
| |-----------------------------------------+
| | action: croak 'Workspace not prepared' |
| |<----------------------------------------+
+---------------------+
|
| prepare_workspace()
| action: make temp directory, copy lib_dir into it,
| remember workspace path and relative file path
v
+---------------------+ generate_mutants() (as above)
| |-----------------------------------------+
| WORKSPACE CLEAN |<----------------------------------------+
| (copy is the same |
| as the original) | run_tests() (as above)
| |-----------------------------------------+
| |<----------------------------------------+
| |
| | prepare_workspace()
| |-----------------------------------------+
| | action: new directory, new clean copy |
| |<----------------------------------------+
+---------------------+
| ^
| | prepare_workspace()
| | action: new directory, new clean copy;
| | the mutated copy is left behind
| apply_mutant(m)
| action: rewrite the workspace copy of the file
| with one mutation
v |
+---------------------+ apply_mutant(m2)
| |-----------------------------------------+
| WORKSPACE MUTATED | action: a second mutation is added on |
| | top of the first (usually a mistake) |
| |<----------------------------------------+
| |
| | generate_mutants(), run_tests()
| |-----------------------------------------+
| | action: as above, state stays the same |
| |<----------------------------------------+
+---------------------+
From any state: when the program ends, every temporary workspace
directory is deleted.
A call that croaks leaves the object in the state it was in. The workspace file is only rewritten after the mutation has succeeded.
No method of this class changes the real project file.
LICENCE AND COPYRIGHT
Copyright 2026 Nigel Horne.
Usage is subject to the terms of GPL2. If you use it, please let me know.