NAME

test-generator-index - Test coverage dashboard generator

DESCRIPTION

test-generator-index generates an HTML test coverage dashboard for publication on GitHub Pages, combining four sources of test quality data into a single report:

• Statement and branch coverage from Devel::Cover, showing which lines and branches were exercised by the test suite.

• LCSAJ path coverage (TER3) from the LCSAJ runtime debugger, showing which control-flow paths were executed. Displayed as blue (covered) or red (uncovered) dots on per-file mutation pages.

• Mutation testing results from bin/test-generator-mutate, showing which injected faults the test suite detected (killed) and which it missed (survived).

• CPAN Testers failure analysis, showing which Perl versions and operating systems are failing, with automatic root cause detection including Perl version cliffs, locale sensitivity, and dependency version cliffs. For each failure report where installed-module data is available, the dashboard also generates a self-contained shell script (cover_html/reproduce/reproduce-GUID.sh) that reproduces the exact environment — pinning every installed module version with cpanm and suggesting the matching Perl via perlbrew/plenv.

In addition to the dashboard, the script drives the mutation-guided test generation pipeline described in "MUTATION-GUIDED TEST GENERATION" in App::Test::Generator. Surviving mutants are used to automatically generate new test stubs and fuzz schemas that target the exact boundary conditions the test suite missed:

• --generate_mutant_tests=DIR produces TODO stub files in DIR/ for all surviving mutants, grouped by source file and deduplicated by line.

• --generate_test=mutant additionally attempts to produce runnable YAML schemas in DIR/conf/ for NUM_BOUNDARY survivors using App::Test::Generator::SchemaExtractor.

• --generate_fuzz augments existing schemas in DIR/conf/ with boundary values from surviving mutants, writing timestamped copies that are picked up automatically by t/fuzz.t.

The script is designed to be shared across projects. Copy it into the scripts/ directory of each project that uses it:

cp ../App-Test-Generator/scripts/test-generator-index scripts/

It is invoked automatically by scripts/generate_test_dashboard on each CI push via .github/workflows/dashboard.yml.

SYNOPSIS

https://$github_user.github.io/$github_repo/coverage/

INPUTS

cover_html/cover.json     - Devel::Cover JSON report (statement/branch/condition)
mutation.json             - Mutation testing results from test-generator-mutate
cover_html/lcsaj_hits.json - LCSAJ path hit data from the LCSAJ runtime debugger
cover_html/mutation_html/lib/ - Per-file LCSAJ path definitions (.lcsaj.json)
coverage_history/*.json   - Historical coverage snapshots for the trend chart

OUTPUTS

cover_html/index.html          - Main dashboard (coverage table, trend chart,
                                 CPAN Testers failures, mutation report)
cover_html/mutation_html/      - Per-file mutation heatmap pages
cover_html/reproduce/          - CPAN Testers reproduction scripts, one per
                                 FAIL report where installed-module data was
                                 available (reproduce-GUID.sh, mode 0755).
                                 Each script pins every installed module
                                 version with cpanm and suggests the matching
                                 Perl via perlbrew/plenv.

OPTIONS

--generate_mutant_tests=DIR
    Generate a timestamped test stub file in DIR (typically 'xt/') for
    surviving mutants. The file is named mutant_YYYYMMDD_HHMMSS.t and
    contains:
      - TODO test stubs for High/Medium difficulty survivors, with
        boundary value suggestions, environment variable hints, and
        the enclosing subroutine name for navigation context
      - Comment-only hints for Low difficulty survivors
    Multiple mutations on the same source line are deduplicated into
    one stub - one good test kills all variants on that line.
    File is skipped entirely if there are no survivors to report.
    If not given, no test stubs are generated.

--generate_test=CLASS
    When combined with --generate_mutant_tests=DIR, attempts to produce
    runnable test artefacts for surviving mutants rather than TODO stubs.

    Currently supported classes:

      mutant  For NUM_BOUNDARY survivors, calls
              App::Test::Generator::SchemaExtractor to extract the schema
              for the enclosing subroutine and augments it with the
              boundary value from the mutant (plus one value either side).
              The resulting YAML schema is written to DIR/conf/ and is
              picked up automatically by t/fuzz.t on the next test run.
              Falls back to a TODO stub if SchemaExtractor fails, the
              enclosing sub cannot be determined, or the extracted schema
              confidence is too low (very_low or none).

    This option is designed to accept additional classes in future, for
    example corpus-driven or property-based test generation.
    If not given, only TODO stubs are produced.

 --generate_fuzz
    Scans t/conf/ for existing YAML schema files and augments copies
    of them with boundary values extracted from surviving NUM_BOUNDARY
    mutants whose enclosing subroutine matches the schema's function
    field. The original schema is never modified. Augmented copies are
    written to xt/conf/mutant_fuzz_YYYYMMDD_HHMMSS_FUNCTION.yml and
    are picked up automatically by t/fuzz.t on the next test run.

    Schemas whose filename already starts with mutant_fuzz_ are skipped
    to avoid augmenting previously augmented schemas. Schemas with no
    matching survivors are skipped (with a note if --verbose is active).
    New boundary values are merged into whichever edge key already
    exists in the schema (edge_case_array or edge_cases), with
    deduplication against existing values.

    This flag is independent of --generate_test and can be used alone.

DASHBOARD SECTIONS

Coverage Table    - Per-file statement/branch/condition/subroutine
                    percentages with delta vs previous snapshot,
                    sortable columns, and sparkline trend per file
Coverage Trend    - Chart of total coverage over recent commits with
                    linear regression line, zoom and pan support
RT Issues         - Count of open RT tickets for the distribution
CPAN Testers      - Failure table for the current release, with
                    Perl version cliff detection, locale analysis,
                    dependency version cliff detection, and root
                    cause confidence scoring
Mutation Report   - Per-file mutation score (killed/survived/total),
                    cyclomatic complexity, and TER3 (LCSAJ path
                    coverage) with raw fraction
Per-file Pages    - Line-by-line mutation heatmap with TER1/TER2/TER3
                    metrics, LCSAJ path markers, and expandable
                    mutant details with suggested tests

DEPENDENCIES

Cwd, File::Basename, File::Glob, File::Path,
File::Slurp, File::Spec, File::stat, Getopt::Long, HTML::Entities,
HTTP::Tiny, IPC::Run3, JSON::MaybeXS, List::Util, POSIX,
Readonly, Time::HiRes, URI::Escape, WWW::RT::CPAN, version

FUNCTIONS

generate_reproduction_script

my $name = generate_reproduction_script($dist, $version, $report, $installed_mods, $outdir);

Generates a self-contained POSIX shell script that reproduces the exact environment of a CPAN Testers failure report, then returns the script's filename (reproduce-GUID.sh) on success, or undef if the report has no guid field.

Arguments

• $dist — distribution name (e.g. Foo-Bar).

• $version — distribution version string (e.g. 1.23).

• $report — hashref with at minimum guid, perl, osname, and reporter keys as returned by the CPAN Testers API.

• $installed_mods — hashref of Module => version pairs from "extract_installed_modules", or an empty hashref.

• $outdir — root output directory. The script is written to $outdir/reproduce/reproduce-$guid.sh (mode 0755).

Output script content

The generated script contains:

• A set -e guard followed by a perlbrew advisory comment.

• If $installed_mods is non-empty: a cpanm --notest command that pins every installed module at its exact failing version (modules are single-quoted and sorted alphabetically to prevent shell-injection).

• A cpanm --look Dist@version command to open the distribution's source, plus a comment showing the local-checkout alternative (perl Makefile.PL && make && make test).

extract_installed_modules

my $mods = extract_installed_modules($html);

Parses the Installed modules: section from a CPAN Testers report HTML page and returns a hashref of Module => version pairs. Returns an empty hashref if $html is undef or the section is absent.

Module names are validated against [A-Z]\w*(?:::\w+)* and versions against [\d._]+, so the returned hashref is safe for shell interpolation.

AUTHOR

Nigel Horne <njh@nigelhorne.com>

cpanm App::Test::Generator

perl -MCPAN -e shell
install App::Test::Generator