NAME

bench.pl - Compare the performance of perl code snippets across multiple perls.

SYNOPSIS

# Basic: run the tests in t/perf/benchmarks against two or
# more perls

bench.pl [options] perlA[=labelA] perlB[=labelB] ...

# run the tests against the same perl twice, with varying options

bench.pl [options] perlA=bigint --args='-Mbigint' perlA=plain

# Run bench on blead, saving results to file; then modify the blead
# binary, and benchmark again, comparing against the saved results

bench.pl [options] --write=blead.time ./perl=blead
# ... hack hack hack, updating ./perl ...
bench.pl --read=blead.time ./perl=hacked

# You can also combine --read with --write and new benchmark runs

bench.pl --read=blead.time --write=last.time -- ./perl=hacked

DESCRIPTION

By default, bench.pl will run code snippets found in t/perf/benchmarks (or similar) under cachegrind, in order to calculate how many instruction reads, data writes, branches, cache misses, etc. that one execution of the snippet uses. Usually it will run them against two or more perl executables and show how much each test has gotten better or worse.

It is modelled on the perlbench tool, but since it measures instruction reads etc., rather than timings, it is much more precise and reproducible. It is also considerably faster, and is capable of running tests in parallel (with -j). Rather than displaying a single relative percentage per test/perl combination, it displays values for 13 different measurements, such as instruction reads, conditional branch misses etc.

There are options to write the raw data to a file, and to read it back. This means that you can view the same run data in different views with different selection and sort options. You can also use this mechanism to save the results of timing one perl, and then read it back while timing a modification, so that you don't have rerun the same tests on the same perl over and over, or have two perl executables built at the same time.

The optional =label after each perl executable is used in the display output. If you are doing a two step benchmark then you should provide a label for at least the "base" perl. If a label isn't specified, it defaults to the name of the perl executable. Labels must be unique across all current executables, plus any previous ones obtained via --read.

In its most general form, the specification of a perl executable is:

path/perl=+mylabel --args='-foo -bar' --args='-baz' \
                   --env='A=a' --env='B=b'

This defines how to run the executable path/perl. It has a label, which due to the +, is appended to the binary name to give a label of path/perl=+mylabel (without the +, the label would be just mylabel).

It can be optionally followed by one or more --args or --env switches, which specify extra command line arguments or environment variables to use when invoking that executable. Each --env switch should be of the form --env=VARIABLE=value. Any --arg values are concatenated to the eventual command line, along with the global --perlargs value if any. The above would cause a system() call looking something like:

PERL_HASH_SEED=0 A=a B=b valgrind --tool=cachegrind \
    path/perl -foo -bar -baz ....

OPTIONS

General options

• --action=foo

  What action to perform. The default is grind, which runs the benchmarks using cachegrind as the back end. The only other action at the moment is selftest, which runs some basic sanity checks and produces TAP output.

• --debug

  Enable debugging output.

• ---help

  Display basic usage information.

• -v --verbose

  Display progress information.

Test selection options

• --tests=FOO

  Specify a subset of tests to run (or in the case of --read, to read). It may be either a comma-separated list of test names, or a regular expression. For example

  --tests=expr::assign::scalar_lex,expr::assign::2list_lex
  --tests=/^expr::/

Input options

• -r file --read=file

  Read in saved data from a previous --write run from the specified file. If --tests is present too, then only tests matching those conditions are read from the file.

  --read may be specified multiple times, in which case the results across all files are aggregated. The list of test names from each file (after filtering by --tests) must be identical across all files.

  This list of tests is used instead of that obtained from the normal benchmark file (or --benchfile) for any benchmarks that are run.

  The perl labels must be unique across all read in test results.

  Requires JSON::PP to be available.

Benchmarking options

Benchmarks will be run for all perls specified on the command line. These options can be used to modify the benchmarking behavior:

• --autolabel

  Generate a unique label for every executable which doesn't have an explicit =label. Works by stripping out common prefixes and suffixes from the executable names, then for any non-unique names, appending -0, -1, etc. text directly surrounding the unique part which look like version numbers (i.e. which match /[0-9\.]+/) aren't stripped. For example,

  perl-5.20.0-threaded  perl-5.22.0-threaded  perl-5.24.0-threaded

  stripped to unique parts would be:

  20  22  24

  but is actually only stripped down to:

  5.20.0  5.22.0  5.24.0

  If the final results are plain integers, they are prefixed with "p" to avoid looking like column numbers to switches like --norm=2.

• --benchfile=foo

  The path of the file which contains the benchmarks (t/perf/benchmarks by default).

• --grindargs=foo

  Optional command-line arguments to pass to all cachegrind invocations.

• -j N --jobs=N

  Run N jobs in parallel (default 1). This determines how many cachegrind process will run at a time, and should generally be set to the number of CPUs available.

• --perlargs=foo

  Optional command-line arguments to pass to every perl executable. This may optionaly be combined with --args switches following individual perls. For example:

  bench.pl --perlargs='-Ilib -It/lib' .... \
      perlA --args='-Mstrict' \
      perlB --args='-Mwarnings'

  would cause the invocations

  perlA -Ilib -It/lib -Mstrict
  perlB -Ilib -It/lib -Mwarnings

Output options

Any results accumulated via --read or by running benchmarks can be output in any or all of these three ways:

• -w file --write=file

  Save the raw data to the specified file. It can be read back later with --read. If combined with --read then the output file will be the merge of the file read and any additional perls added on the command line.

  Requires JSON::PP to be available.

• --bisect=field,minval,maxval

  Exit with a zero status if the named field is in the specified range; exit with 1 otherwise. It will complain if more than one test or perl has been specified. It is intended to be called as part of a bisect run, to determine when something changed. For example,

  bench.pl -j 8 --tests=foo --bisect=Ir,100,105 --perlargs=-Ilib \
      ./miniperl

  might be called from bisect to find when the number of instruction reads for test foo falls outside the range 100..105.

• --show

  Display the results to stdout in human-readable form. This is enabled by default, except with --write and --bisect. The following sub-options alter how --show behaves.

  • --average

    Only display the overall average, rather than the results for each individual test.

  • --compact=perl

    Display the results for a single perl executable in a compact form. Which perl to display is specified in the same manner as --norm.

  • --fields=a,b,c

    Display only the specified fields; for example,

    --fields=Ir,Ir_m,Ir_mm

    If only one field is selected, the output is in more compact form.

  • --norm=foo

    Specify which perl column in the output to treat as the 100% norm. It may be:

    * a column number (0..N-1),

    * a negative column number (-1..-N) which counts from the right (so -1 is the right-most column),

    * or a perl executable name,

    * or a perl executable label.

    It defaults to the leftmost column.

  • --raw

    Display raw data counts rather than percentages in the outputs. This allows you to see the exact number of intruction reads, branch misses etc. for each test/perl combination. It also causes the AVERAGE display per field to be calculated based on the average of each tests's count rather than average of each percentage. This means that tests with very high counts will dominate.

  • --sort=field:perl

    Order the tests in the output based on the value of field in the column perl. The perl value is as per --norm. For example

    bench.pl --sort=Dw:perl-5.20.0 \
        perl-5.16.0 perl-5.18.0 perl-5.20.0