NAME

App::Test::Generator::Mutator - Generate and apply mutation tests

VERSION

Version 0.49

SYNOPSIS

A "mutant" is a copy of your code with one small, deliberate mistake. If your tests still pass, the mutant "survived" and your tests missed the mistake. If a test fails, the mutant was "killed", which is good.

Pattern 1: list the mutants for a file. Nothing is changed on disk.

use App::Test::Generator::Mutator;

my $mutator = App::Test::Generator::Mutator->new(
    file    => 'lib/My/Module.pm',
    lib_dir => 'lib',
);

my $mutants = $mutator->generate_mutants();
printf "Generated %d mutants\n", scalar @{$mutants};
printf "line %d: %s\n", $_->line, $_->description for @{$mutants};

Pattern 2: test every mutant. Run this from the top directory of the project. The mutated file is copied over the real file for the test run and the real file is always put back. This is what bin/test-generator-mutate does.

my $file = 'lib/My/Module.pm';
my $read = sub { open my $fh, '<', $_[0]; local $/; <$fh> };
my $original = $read->($file);

for my $m (@{$mutants}) {
    my $workspace = $mutator->prepare_workspace();    # clean copy each time
    $mutator->apply_mutant($m);
    my $mutated = $read->("$workspace/$file");

    { open my $fh, '>', $file; print $fh $mutated }
    my $survived = $mutator->run_tests();
    { open my $fh, '>', $file; print $fh $original }

    printf "%s: %s\n", ($survived ? 'SURVIVED' : 'KILLED'), $m->description;
}

Pattern 3: use fast mode to get fewer mutants and a shorter run.

my $quick = App::Test::Generator::Mutator->new(
    file           => 'lib/My/Module.pm',
    mutation_level => 'fast',
);
my $fewer = $quick->generate_mutants();
printf "fast: %d mutants, full: %d mutants\n",
    scalar @{$fewer}, scalar @{$mutants};

Pattern 4: protect lines of your own code from mutation. Put these two comment lines around them in the file to be mutated. Each marker must be alone on its line.

## MUTANT_SKIP_BEGIN
return 1 if $n <= 1;    # no mutant will be made for this line
## MUTANT_SKIP_END

DESCRIPTION

App::Test::Generator::Mutator is a mutation engine that programmatically alters Perl source files to evaluate the effectiveness of a project's test suite. It analyses modules, generates systematic code mutations (such as conditional inversions, logical operator changes, and numeric boundary flips), and applies them within an isolated workspace so tests can be executed safely against each modified variant.

By tracking which mutants are killed (cause tests to fail) versus those that survive (tests still pass), the module enables calculation of a mutation score, providing a quantitative measure of how well the test suite detects unintended behavioural changes.

new

Construct a new Mutator for a given source file.

my $mutator = App::Test::Generator::Mutator->new(
    file           => 'lib/My/Module.pm',
    lib_dir        => 'lib',
    mutation_level => 'full',
);

Arguments

  • file

    Path to the Perl source file to mutate. Required. Must exist on disk.

  • lib_dir

    Root library directory. Optional - defaults to lib. Use a relative path. file must begin with this path.

  • mutation_level

    Controls the breadth of mutation. full applies all mutations; fast deduplicates and removes redundant mutants first. Optional - defaults to full.

Returns

A blessed hashref. Croaks if file is missing or does not exist. The file is only checked here. It is read later, by generate_mutants.

API SPECIFICATION

input

{
    file           => { type => SCALAR },
    lib_dir        => { type => SCALAR, optional => 1 },
    mutation_level => { type => SCALAR, optional => 1 },
}

output

{
    type => OBJECT,
    isa  => 'App::Test::Generator::Mutator',
}

EXAMPLE

my $m = App::Test::Generator::Mutator->new(
    file           => 'lib/Acme/Widget.pm',
    mutation_level => 'fast',
);

MESSAGES

file required

file was not supplied.

file not found: PATH

file was supplied but does not exist on disk.

The mathematical description is in "FORMAL SPECIFICATION".

generate_mutants

Parse the target file and generate all mutants by running each registered mutation strategy against the PPI document.

my $mutants = $mutator->generate_mutants();   # scalar context: arrayref
for my $m (@{$mutants}) { ... }

my @mutants = $mutator->generate_mutants();   # list context: flat list (backward-compat)

Arguments

None beyond $self.

Returns

An arrayref of App::Test::Generator::Mutant objects. In fast mode, redundant and duplicate mutants are removed before returning. Lines within ## MUTANT_SKIP_BEGIN / ## MUTANT_SKIP_END annotation blocks are excluded from the candidate list entirely. After this method returns, $self->{skip_lines} contains a hashref mapping excluded line numbers to 1.

API SPECIFICATION

input

{
    self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
}

output

{
    type     => ARRAYREF,
    elements => { type => OBJECT, isa => 'App::Test::Generator::Mutant' },
}

EXAMPLE

my $mutants = $mutator->generate_mutants();
printf "%d mutants generated\n", scalar @{$mutants};

for my $m (@{$mutants}) {
    printf "line %d: %s\n", $m->line, $m->description;
}

MESSAGES

Unable to parse FILE

PPI could not parse the source file (syntax error or unreadable file). FILE is the path passed to new.

FILE: MUTANT_SKIP_BEGIN at line N with no prior MUTANT_SKIP_END

A ## MUTANT_SKIP_BEGIN marker was found while already inside a skip block.

FILE: MUTANT_SKIP_END at line N with no matching MUTANT_SKIP_BEGIN

A ## MUTANT_SKIP_END marker was found with no preceding ## MUTANT_SKIP_BEGIN.

FILE: MUTANT_SKIP_BEGIN at line N has no matching MUTANT_SKIP_END

The source file ended while still inside a skip block.

The mathematical description is in "FORMAL SPECIFICATION".

PSEUDOCODE

parse file with PPI
scan lines for MUTANT_SKIP_BEGIN / MUTANT_SKIP_END pairs, giving skip_lines
for each registered mutation strategy:
    skip if strategy does not apply_to(doc)
    for each mutant from strategy->mutate(doc):
        include unless mutant.line is in skip_lines
if mutation_level == 'fast':
    deduplicate and remove redundant mutants
return arrayref (or flat list in list context)

prepare_workspace

Prepare an isolated temporary workspace for a single mutant.

The entire lib_dir tree is copied into the workspace so that all module dependencies resolve correctly when the test suite runs against the mutant. Only after this copy is complete is the single target file overwritten by apply_mutant.

my $workspace = $mutator->prepare_workspace();
$mutator->apply_mutant($mutant);
local $ENV{PERL5LIB} = "$workspace/lib";
my $survived = (system('prove', 't') == 0);

Arguments

None beyond $self.

Returns

A string containing the absolute path to the temporary directory created. The directory is removed automatically when the program ends (File::Temp's CLEANUP => 1 behaviour). It is not removed earlier.

Side effects

Creates a temporary directory. Recursively copies lib_dir into it. Sets $self->{_workspace}, $self->{_relative}, and $self->{_lib_basename}. Does not modify $self->{lib_dir}.

Notes

Call prepare_workspace again before each apply_mutant call. apply_mutant does not undo the previous mutant, so two mutants applied to the same workspace are both present in the file.

Each call makes a new directory and a new full copy of lib_dir. The object then points at the new one. Older workspaces stay on disk until the program ends, so a long run needs disk space for one copy per call.

API SPECIFICATION

input

{
    self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
}

output

{
    type => SCALAR,
}

EXAMPLE

my $workspace = $mutator->prepare_workspace();
# workspace is an absolute temp dir path
# original lib_dir value is unchanged
printf "original lib_dir still: %s\n", $mutator->{lib_dir};

MESSAGES

dircopy failed: $!

The lib_dir tree could not be copied into the temporary workspace directory, usually a permissions error.

The mathematical description is in "FORMAL SPECIFICATION".

apply_mutant

Apply a single mutant's transform to the target file in the workspace.

$mutator->apply_mutant($mutant);

Arguments

Returns

Nothing useful. Do not test the return value. Modifies the workspace copy of the target file in place.

Side effects

Overwrites the target file in the workspace with the mutated version. The real project file is never touched by this method.

API SPECIFICATION

input

{
    self   => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
    mutant => { type => OBJECT, isa => 'App::Test::Generator::Mutant'  },
}

output

{ type => UNDEF }

EXAMPLE

for my $m (@{$mutants}) {
    my $workspace = $mutator->prepare_workspace();
    $mutator->apply_mutant($m);
    # "$workspace/lib/My/Module.pm" now holds exactly one mutation
}

MESSAGES

Workspace not prepared -- call prepare_workspace first

apply_mutant was called before prepare_workspace.

Relative path not set -- call prepare_workspace first

Internal: the relative-path field was not set by prepare_workspace.

Failed to parse TARGET

PPI could not parse the workspace copy of the target file.

Can't call method "transform" on an undefined value

A Perl error, not a message from this module. $mutant was undef.

The mathematical description is in "FORMAL SPECIFICATION".

run_tests

Run the project's test suite and return whether all tests passed.

my $survived = $mutator->run_tests();

Arguments

None beyond $self.

Returns

A true value (1) if all tests passed (mutant survived). A false value (the empty string, not 0) if any test failed or prove could not be started (mutant killed). Test it with if, not with == 0.

Side effects

Executes an external process running the test suite. The output of prove goes to the terminal.

Notes

Runs prove -l t in the current directory. This tests the lib directory of the project, not the workspace. To test a mutant, copy the mutated file from the workspace over the project file first, and put the original back afterwards. See Pattern 2 in the "SYNOPSIS".

The prove program is looked for next to the running perl first, then in Perl's own bin directory, and only then on PATH.

There is no time limit. A mutant that causes an endless loop makes this call wait for ever.

API SPECIFICATION

input

{
    self => { type => OBJECT, isa => 'App::Test::Generator::Mutator' },
}

output

{ type => SCALAR }

EXAMPLE

my $survived = $mutator->run_tests();
if($survived) {
    print "mutant survived\n";
} else {
    print "mutant killed\n";
}

The mathematical description is in "FORMAL SPECIFICATION".

COMMON PITFALLS

Calling apply_mutant before prepare_workspace

apply_mutant requires prepare_workspace to have been called first. The workspace holds the isolated copy of lib/ that receives mutations.

Passing an absolute path as lib_dir

lib_dir must be a relative path (e.g. lib). An absolute path causes apply_mutant to construct a doubled directory under the workspace and then fail to find the target file.

Checking $mutator->{workspace} after the refactor

Internal workspace state is stored in _workspace, _relative, and _lib_basename (note the underscore prefix). The original lib_dir value is never overwritten. Old code that reads $mutator->{workspace} or $mutator->{relative} (without underscores) will not find these keys.

Forgetting that run_tests drives prove from $^X

run_tests resolves prove from the same Perl binary used to run the script. If you shell out to prove directly in your own integration code, make sure you are using the matching prove.

generate_mutants in list vs scalar context

generate_mutants returns a flat list in list context and an arrayref in scalar context. Assign to an arrayref (my $m = $mutator->generate_mutants()) to guarantee you always get a reference regardless of calling context.

Applying two mutants to one workspace

apply_mutant changes the workspace file and leaves it changed. If you apply a second mutant without a new prepare_workspace call, the file holds both mutations and your results are wrong.

Expecting run_tests to test the workspace

run_tests runs the tests against the project's own lib directory. If you only call apply_mutant and then run_tests, the tests load the original code, every mutant "survives", and the kill rate is 0%.

An unknown mutation_level is not an error

Only the exact text fast turns on fast mode. Any other value, for example Fast or quick, is accepted silently and works like full.

How undef and other false values are handled
  • file => undef makes new croak with file required. An empty string makes it croak with file not found.

  • lib_dir and mutation_level use their default when the value is undef, an empty string or 0. You cannot have a library directory that is really named 0.

  • apply_mutant(undef) stops with a Perl error, after the workspace check.

  • run_tests returns an empty string for "failed". It never returns undef.

  • A mutant's line, original or description may be undef. Fast mode treats each missing value as an empty string when it looks for duplicates.

Extra arguments to new are ignored

A wrong spelling such as libdir or level is not reported. The default value is used and no warning is given.

Skip markers must be alone on their line

## MUTANT_SKIP_BEGIN and ## MUTANT_SKIP_END are only seen when nothing else except spaces is on the same line. Markers cannot be nested. The two marker lines themselves are also skipped.

ENCODING

  • The file and lib_dir arguments

    Paths are passed to the operating system unchanged. ASCII paths always work. For non-ASCII directory or file names, pass the path as bytes in the encoding of your file system (normally UTF-8).

  • Source file contents

    The file to mutate is read and written by PPI as raw bytes. UTF-8 text, other non-ASCII text and emojis inside strings, comments and POD are copied to the mutated file byte for byte. They are not damaged and they do not need to be decoded first.

  • Text returned by a mutant

    $mutant->original, $mutant->description and $mutant->line_content can contain pieces of your source code. These are byte strings. If the source is UTF-8 and you print them through a UTF-8 file handle, decode them first with Encode::decode('UTF-8', $text).

  • Skip markers

    The markers are ASCII. They are found correctly in a file of any ASCII-compatible encoding, including UTF-8.

  • Error messages

    Messages contain the file path exactly as you gave it.

LIMITATIONS

  • Single strategy registry

    The four built-in mutation strategies are hardcoded in new. There is no plugin mechanism for registering additional strategies without subclassing. A future version should accept a strategies arrayref argument.

  • No parallelism

    run_tests is synchronous. For large test suites or large mutant sets, wall-clock time scales linearly. The in-place mutation strategy also serialises all mutants behind a single file lock.

  • PPI re-parse per apply_mutant call

    apply_mutant re-parses the workspace copy of the target file for every mutant. For very large single-file modules the PPI parse time may dominate.

  • apply_mutant does not restore on abnormal exit

    If the process is killed between the write and restore in bin/test-generator-mutate, the project file is left mutated. A git restore lib/... recovers it.

SEE ALSO

bin/test-generator-mutate
Devel::Mutator

AUTHOR

Nigel Horne, <njh at nigelhorne.com>

FORMAL SPECIFICATION

This section describes each method in Z notation. It is for readers who want an exact, mathematical statement. You do not need it to use the module.

How to read it: a name ending in ? is an input, a name ending in ! is an output, and a name ending in ' is the value after the call. Δ before the state name means "this call may change the object". Ξ means "this call does not change the object".

LEVEL == { 'full', 'fast' }

┌─ Mutator ─────────────────────────────────────────
│ file, lib_dir : PATH
│ mutation_level : STRING
│ skip_lines : ℙ ℕ₁ ∪ { undef }
│ workspace : PATH ∪ { undef }
│ relative : PATH ∪ { undef }
├───────────────────────────────────────────────────
│ workspace = undef ⇔ relative = undef
└───────────────────────────────────────────────────

new

┌─ New ─────────────────────────────────────────────
│ Mutator′
│ file? : VALUE
│ lib_dir?, mutation_level? : VALUE
│ result! : Mutator
├───────────────────────────────────────────────────
│ pre   file? ≠ undef ∧ file? names an existing plain file
│ post  file′ = file?
│       lib_dir′ = if lib_dir? is true
│                  then lib_dir? else 'lib'
│       mutation_level′ = if mutation_level? is true
│                         then mutation_level? else 'full'
│       skip_lines′ = undef ∧ workspace′ = undef ∧ relative′ = undef
│       result! = θ Mutator′
└───────────────────────────────────────────────────

file? = undef                  ⇒ croak 'file required'
file? is not an existing file  ⇒ croak 'file not found: ' ⁀ file?

generate_mutants

┌─ GenerateMutants ─────────────────────────────────
│ Δ Mutator
│ result! : seq Mutant
│ all : seq Mutant
├───────────────────────────────────────────────────
│ pre   file is valid Perl that PPI can parse
│       skip markers in file are paired and not nested
│ post  skip_lines′ = { n : ℕ₁ | line n of file lies between
│                       a BEGIN marker and its END marker,
│                       both marker lines included }
│       all = every mutant that the four strategies
│             produce for file, in strategy order
│       ∀ m : ran result! • m ∈ ran all ∧ m.line ¬in; skip_lines′
│       mutation_level ≠ 'fast' ⇒
│           result! = all with skipped lines removed
│       mutation_level = 'fast' ⇒
│           (∀ i, j : dom result! • i ≠ j ⇒
│               key(result!(i)) ≠ key(result!(j)))
│           ∧ (∀ m : ran result! • ¬ redundant(m))
│       file′ = file ∧ lib_dir′ = lib_dir
│       workspace′ = workspace ∧ relative′ = relative
└───────────────────────────────────────────────────

key(m) == (m.line, m.original, m.description)

In list context the members of result! are returned as a list.

prepare_workspace

┌─ PrepareWorkspace ────────────────────────────────
│ Δ Mutator
│ result! : PATH
├───────────────────────────────────────────────────
│ pre   lib_dir names a directory that can be copied
│ post  result! names a directory created by this call
│       result! ≠ workspace
│       result!/basename(lib_dir) is a full copy of lib_dir
│       workspace′ = result!
│       relative′ = file with the leading lib_dir removed
│       file′ = file ∧ lib_dir′ = lib_dir
│       mutation_level′ = mutation_level
│       skip_lines′ = skip_lines
└───────────────────────────────────────────────────

the copy fails  ⇒ croak 'dircopy failed: ' ⁀ reason

apply_mutant

┌─ ApplyMutant ─────────────────────────────────────
│ Ξ Mutator
│ mutant? : Mutant
│ target : PATH
├───────────────────────────────────────────────────
│ pre   workspace ≠ undef ∧ relative is true
│       mutant?.transform is a CODE reference
│       target = workspace/basename(lib_dir)/relative
│       target is valid Perl that PPI can parse
│ post  content′(target) = mutant?.transform(content(target))
│       content′(file) = content(file)
└───────────────────────────────────────────────────

workspace = undef  ⇒ croak 'Workspace not prepared -- ...'
relative is false  ⇒ croak 'Relative path not set -- ...'
target not parsed  ⇒ croak 'Failed to parse ' ⁀ target

run_tests

┌─ RunTests ────────────────────────────────────────
│ Ξ Mutator
│ result! : { 1, '' }
├───────────────────────────────────────────────────
│ post  result! = 1 ⇔ the command 'prove -l t', run in the
│                     current directory, exits with status 0
└───────────────────────────────────────────────────

STATE DIAGRAM

The object has three states. The state says what has been prepared so far.

    new(file => undef) or new(file => missing file)
  +--------------------------------------------------+
  |  action: croak, no object                        |
  v                                                  |
+-------------+                                      |
|  NO OBJECT  |--------------------------------------+
+-------------+
      |
      | new(file => existing file)
      | action: store file, lib_dir, mutation_level;
      |         build the four mutation strategies
      v
+---------------------+  generate_mutants()
|                     |-----------------------------------------+
|     CONSTRUCTED     |  action: read file, set skip_lines,     |
|  (no workspace yet) |  return the mutants (state stays same)  |
|                     |<----------------------------------------+
|                     |
|                     |  run_tests()
|                     |-----------------------------------------+
|                     |  action: run 'prove -l t', return 1/''  |
|                     |<----------------------------------------+
|                     |
|                     |  apply_mutant(m)
|                     |-----------------------------------------+
|                     |  action: croak 'Workspace not prepared' |
|                     |<----------------------------------------+
+---------------------+
      |
      | prepare_workspace()
      | action: make temp directory, copy lib_dir into it,
      |         remember workspace path and relative file path
      v
+---------------------+  generate_mutants()   (as above)
|                     |-----------------------------------------+
|   WORKSPACE CLEAN   |<----------------------------------------+
|  (copy is the same  |
|   as the original)  |  run_tests()          (as above)
|                     |-----------------------------------------+
|                     |<----------------------------------------+
|                     |
|                     |  prepare_workspace()
|                     |-----------------------------------------+
|                     |  action: new directory, new clean copy  |
|                     |<----------------------------------------+
+---------------------+
      |            ^
      |            | prepare_workspace()
      |            | action: new directory, new clean copy;
      |            |         the mutated copy is left behind
      | apply_mutant(m)
      | action: rewrite the workspace copy of the file
      |         with one mutation
      v            |
+---------------------+  apply_mutant(m2)
|                     |-----------------------------------------+
|  WORKSPACE MUTATED  |  action: a second mutation is added on  |
|                     |  top of the first (usually a mistake)   |
|                     |<----------------------------------------+
|                     |
|                     |  generate_mutants(), run_tests()
|                     |-----------------------------------------+
|                     |  action: as above, state stays the same |
|                     |<----------------------------------------+
+---------------------+

From any state: when the program ends, every temporary workspace
directory is deleted.

A call that croaks leaves the object in the state it was in. The workspace file is only rewritten after the mutation has succeeded.

No method of this class changes the real project file.

LICENCE AND COPYRIGHT

Copyright 2026 Nigel Horne.

Usage is subject to the terms of GPL2. If you use it, please let me know.