NAME

fuzz-harness-generator - Generate fuzzing + corpus-based test harnesses from test schemas

SYNOPSIS

fuzz-harness-generator [-r] [--strict-documented] [-o output_file] input.yaml
fuzz-harness-generator --mutation-results mutation.json -o t/fuzz.t input.yaml
fuzz-harness-generator --dry-run input.yaml
fuzz-harness-generator --replay-corpus schemas/corpus/ -o t/fuzz_replay.t
fuzz-harness-generator --replay-corpus schemas/corpus/translate.json -o t/fuzz_replay.t

DESCRIPTION

This tool generates a test file that fuzzes and validates a target module's function or method, using both randomized fuzz cases and a static corpus cases (Perl or YAML).

It can also generate regression test files from corpus JSON files previously written by extract-schemas --fuzz, using --replay-corpus.

A starter input.yaml can be created using extract-schemas which is also in this package.

OPTIONS

--help

Show this help.

--input

The input configuration file

--output

The (optional) output file.

--dry-run

Validate the input configuration and schema extraction without writing any output files or running tests.

--run

Call prove on the output file.

fuzz-harness-generator -r t/conf/data_text_append.conf will, therefore, dynamically create and run tests on the append method of Data::Text

--strict-documented

Fail the generated harness when fuzzing finds a routine that contradicts a POD-derived constraint: a bound, a memberof list or a matches pattern. By default the harness runs in advisory mode: an outside value is still tried, and a routine that accepts it is reported as accepting a value its POD says is invalid, but the run does not fail. This option makes each recorded contradiction a failing test.

--mutation-results FILE

Feed surviving mutants back into harness generation. FILE is the JSON report written by test-generator-mutate --json. A surviving numeric-boundary mutant is traced to the comparison it mutated, and that comparison becomes a min or max constraint on the matching input field, so the generated harness exercises the boundary the test suite missed. Existing constraints are never replaced, and survivors that do not describe a numeric comparison are listed for review. Only survivors belonging to the schema's module are used.

--replay-corpus PATH

Instead of generating a fuzz harness, generate a regression test file from one or more corpus JSON files previously written by extract-schemas --fuzz.

PATH may be either:

  • A single corpus file, e.g. schemas/corpus/translate.json

  • A directory, e.g. schemas/corpus/ - all *.json files in that directory will be included

The generated test file contains one failing test per bug recorded in the corpus. Each test calls the target method with the exact input that previously caused a crash and expects it not to die. Tests will be red until the underlying bug is fixed, at which point they go green and stay green - acting as permanent regression tests.

Only corpus entries with recorded bugs are included. Clean corpus entries (inputs that did not cause a bug) are ignored.

--version

Prints the version of App::Test::Generator

REPLAY CORPUS WORKFLOW

Step 1: discover bugs with extract-schemas

extract-schemas --fuzz lib/MyModule.pm

This runs coverage-guided fuzzing and writes any discovered bugs to schemas/corpus/method.json.

Step 2: generate a regression test file

fuzz-harness-generator --replay-corpus schemas/corpus/ -o t/fuzz_replay.t

This reads every *.json file in schemas/corpus/, extracts the recorded bugs, and generates a t/fuzz_replay.t that calls each buggy input and expects it not to die.

Step 3: run the regression tests

prove -l t/fuzz_replay.t

Tests will be red for each unfixed bug. Fix the underlying code, re-run prove, and the tests go green.

Step 4: commit both files

Commit schemas/corpus/ and t/fuzz_replay.t to version control. The corpus ensures future fuzz runs build on past discoveries; the replay test ensures fixed bugs stay fixed.

Keeping the replay file up to date

Re-run fuzz-harness-generator --replay-corpus whenever new bugs are discovered or old ones are fixed and removed from the corpus. The generated file should not be edited by hand.

SEE ALSO

App::Test::Generator, App::Test::Generator::CoverageGuidedFuzzer, extract-schemas

AUTHOR

Nigel Horne