NAME
OptreeCheck - check optrees as rendered by B::Concise
SYNOPSIS
OptreeCheck supports regression testing of perl's parser, optimizer, bytecode generator, via a single function: checkOptree(%args). It invokes B::Concise upon sample code, and checks that it 'agrees' with reference renderings.
checkOptree (
name => "test-name', # optional, (synth from others)
# 2 kinds of code-under-test: must provide 1
code => sub {my $a}, # coderef, or source (wrapped and evald)
prog => 'sort @a', # run in subprocess, aka -MO=Concise
bcopts => '-exec', # $opt or \@opts, passed to BC::compile
# errs => '.*', # match against any emitted errs, -w warnings
# skip => 1, # skips test
# todo => 'excuse', # anticipated failures
# fail => 1 # force fail (by redirecting result)
# debug => 1, # turns on regex debug for match test !!
# retry => 1 # retry with debug on test failure
expect => <<'EOT_EOT', expect_nt => <<'EONT_EONT' );
# 1 <;> nextstate(main 45 optree.t:23) v
# 2 <0> padsv[$a:45,46] M/LVINTRO
# 3 <1> leavesub[1 ref] K/REFC,1
EOT_EOT
# 1 <;> nextstate(main 45 optree.t:23) v
# 2 <0> padsv[$a:45,46] M/LVINTRO
# 3 <1> leavesub[1 ref] K/REFC,1
EONT_EONTcheckOptree(%in) Overview
optreeCheck() calls getRendering(), which runs code or prog through B::Concise, and captures its rendering.
It then calls mkCheckRex() to produce a regex which will match the expected rendering, and fail when it doesn't match.
Finally, it compares the 2; like($rendering,/$regex/,$testname).
checkOptree(%Args) API
Accepts %Args, with following requirements and actions:
Either code or prog must be present. prog is some source code, and is passed through via test.pl:runperl, to B::Concise like this: (bcopts are fixed up for cmdline)
'./perl -w -MO=Concise,$bcopts_massaged -e $src'code is a subref, or $src, like above. If it's not a subref, it's treated like source-code, is wrapped as a subroutine, and is passed to B::Concise::compile().
$subref = eval "sub{$src}";
B::Concise::compile($subref).expect and expect_nt are the reference optree renderings. Theyre required, except when the code/prog compilation fails.
I suppose I should also explain these more, but they seem obvious.
# prog => 'sort @a', # run in subprocess, aka -MO=Concise
# noanchors => 1, # no /^$/. needed for 1-liners like above
# skip => 1, # skips test
# todo => 'excuse', # anticipated failures
# fail => 1 # fails (by redirecting result)
# debug => 1, # turns on regex debug for match test !!
# retry => 1 # retry with debug on test failureTest Philosophy
2 platforms --> 2 reftexts: You want an accurate test, independent of which platform you're on. So, two refdata properties, 'expect' and 'expect_nt', carry renderings taken from threaded and non-threaded builds. This has several benefits:
1. native reference data allows closer matching by regex.
2. samples can be eyeballed to grok t-nt differences.
3. data can help to validate mkCheckRex() operation.
4. can develop regexes which accomodate t-nt differences.
5. can test with both native and cross+converted regexes.Cross-testing (expect_nt on threaded, expect on non-threaded) exposes differences in B::Concise output, so mkCheckRex has code to do some cross-test manipulations. This area needs more work.
Test Modes
One consequence of a single-function API is difficulty controlling test-mode. Ive chosen for now to use a package hash, %gOpts, to store test-state. These properties alter checkOptree() function, either short-circuiting to selftest, or running a loop that runs the testcase 2^N times, varying conditions each time. (current N is 2 only).
So Test-mode is controlled with cmdline args, also called options below. Run with 'help' to see the test-state, and how to change it.
selftest
This argument invokes runSelftest(), which tests a regex against the reference renderings that they're made from. Failure of a regex match its 'mold' is a strong indicator that mkCheckRex is buggy.
That said, selftest mode currently runs a cross-test too, they're not completely orthogonal yet. See below.
testmode=cross
Cross-testing is purposely creating a T-NT mismatch, looking at the fallout, and tweaking the regex to deal with it. Thus tests lead to 'provably' complete understanding of the differences.
The tweaking appears contrary to the 2-refs philosophy, but the tweaks will be made in conversion-specific code, which (will) handles T->NT and NT->T separately. The tweaking is incomplete.
A reasonable 1st step is to add tags to indicate when TonNT or NTonT is known to fail. This needs an option to force failure, so the test.pl reporting mechanics show results to aid the user.
testmode=native
This is normal mode. Other valid values are: native, cross, both.
checkOptree Notes
Accepts test code, renders its optree using B::Concise, and matches that rendering against a regex built from one of 2 reference-renderings %in data.
The regex is built by mkCheckRex(\%in), which scrubs %in data to remove match-irrelevancies, such as (args) and [args]. For example, it strips leading '# ', making it easy to cut-paste new tests into your test-file, run it, and cut-paste actual results into place. You then retest and reedit until all 'errors' are gone. (now make sure you haven't 'enshrined' a bug).
name: The test name. May be augmented by a label, which is built from important params, and which helps keep names in sync with whats being tested.'
mkCheckRex
mkCheckRex receives the full testcase object, and constructs a regex. 1st, it selects a reftxt from either the expect or expect_nt items.
Once selected, the reftext is massaged & converted into a Regex that accepts 'good' concise renderings, with appropriate input variations, but is otherwise as strict as possible. For example, it should *not* match when opcode flags change, or when optimizations convert an op to an ex-op.
selection is driven by platform mostly, but also by test-mode, which rather complicates the code. this is worsened by the potential need to make platform specific conversions on the reftext.
match criteria
Opcode arguments (text within braces) are disregarded for matching purposes. This loses some info in 'add[t5]', but greatly simplifys matching 'nextstate(main 22 (eval 10):1)'. Besides, we are testing for regressions, not for complete accuracy.
The regex is anchored by default, but can be suppressed with 'noanchors', allowing 1-liner tests to succeed if opcode is found.
TEST DEVELOPMENT SUPPORT
This optree regression testing framework needs tests in order to find bugs. To that end, OptreeCheck has support for developing new tests, according to the following model:
1. write a set of sample code into a single file, one per
paragraph. f_map and f_sort in ext/B/t/ are examples.
2. run OptreeCheck as a program on the file
./perl -Ilib ext/B/t/OptreeCheck.pm -w ext/B/t/f_map
./perl -Ilib ext/B/t/OptreeCheck.pm -w ext/B/t/f_sort
gentest reads the sample code, runs each to generate a reference
rendering, folds this rendering into an optreeCheck() statement,
and prints it to stdout.
3. run the output file as above, redirect to files, then rerun on
same build (for sanity check), and on thread-opposite build. With
editor in 1 window, and cmd in other, it's fairly easy to cut-paste
the gots into the expects, easier than running step 2 on both
builds then trying to sdiff them together.TODO
There's a considerable amount of cruft in the whole arg-handling setup. I'll replace / strip it before 5.10
Treat %in as a test object, interwork better with Test::*
Refactor mkCheckRex() and selfTest() to isolate the selftest, crosstest, etc selection mechanics.
improve retry, retrydbg, esp. it's control of eval "use re debug". This seems to work part of the time, but isn't stable enough.
CAVEATS
This code is purely for testing core. While checkOptree feels flexible enough to be stable, the whole selftest framework is subject to change w/o notice.