/ 
B-Generate-1.16
River stage two • 9 direct dependents • 13 total dependents
/ BTest

NAME

BTest - helper for testing B module

SYNOPSIS

This module supports easy testing/torture of B module with a 2 layer API. The low-level function provides power, extensibility, and rope, the high-level function provides ease of use by deriving OPs to test from the test program itself.

# Low Level API
# test a single op by calling %tests keys as methods,
# and comparing returned value to the stored value
testop($op, %tests);

# High Level API
# test ops in main_start
test_self_ops(-v => 2);

PURPOSE

B is a tricky yet foundational module which allows a sophisticated user to inspect the optree of a perl program. B is difficult to use correctly, and prone to fatal errors otherwize. It is also undertested.

BTest's goals are to make -

- it easy to do cursory testing of B on ops
- it possible to do deeper tests (inspect op args)
- easy stress testing
- the test output self explanatory (tutorial value)
- the tests look like specifications
- tests are extensible
- it easier to learn how to use B well
- framework gives place to hang further knowledge/kludges

DESCRIPTION

BTest has a 2 level API which gives simplicity (test_self_ops) and power (testop) in support of testing the B module.

test_self_ops(%args)

This function extracts the optree of the main program, and uses a B::Concise rendering of it to determine runnable tests for each op, and then run those tests. This provides a baseline of coverage and test sophistication which can improve over time.

It should be called from a file-under-test, immediately after some main-code (ie not subroutine code).

NOTES:

Currently, we throw away any branches rendered by B::Concise, since they are not part of the exec-order OP-vector. Later, it may become obvious how to use them.

testop($op, %tests)

testop examines $op, based upon user defined %tests. Each key (aside from special ones..) is invoked as an OP method $op->$k(), and the result is validated against $tests{$k}, in a manner based upon the value's type:

scalar - is()
regexp - like()
array  - grep { $retval eq $_ } @array
undef  - call $op->$k(), dont test retval
CODE   - call $code->($op->$k()), let code validate retval

This gives brevity and flexibilty, letting users run methods on one op that would be inappropriate (or even segv!) on another. Consider:

testop($op,
   bcons   => 'a  <$> const[PV "1st thing in main"] sM',
   name    => 'const',
   flags   => &B::OPf_WANT_SCALAR | &B::OPf_MOD, # 32+2
   private => 0,
   type    => 5,
   sv      => sub { '->PV "'.(shift)->PV .'"' },
);

Here, all keys (except bcons) are legitimate methods on $op, they're each run and result is validated against values, typically something like this:

# f  <$> const[PV "1st thing in main"] sM
ok 99 - isa B::SVOP=SCALAR(0x8a32eec)
ok 100 - const->flags is 34
ok 101 - const->name is const
ok 102 - const->private is 0
ok 103 - const->sv()->PV "1st thing in main"
ok 104 - const->type is 5

Some observations:

The bcons param is issued as a diagnostic message prior to the tests being run. This groups the tests visually.

Tests 100+ take the name param and use it as part of the test description, giving an appearance of symbolic representation.

Test 99 gives "$op" rather than ref $op. This prints the op's address too, allowing user/tester to inspect linkage between ops.

One other thing not evident here - the test descriptions are constructed from actual values returned by method calls. This can be confusing when the test fails and the description suggests otherwize. However, doing so makes the descriptions more expressive and informative when the test passes (the normal case).

sv => sub { '->PV "'.(shift)->PV .'"' },

Due to presence of this pair, testop() invokes $code->($op->sv);. This particular callback fetches the string value, prefixed with explanatory text (see above).

You have a lot of flexibility here; you can call any and all methods of the object returned by the $op->method ('sv' here). and you can

- and enough rope to hang yourself.
If the constant was an integer, then ->PV would probably be bad.

NB: this sv example was developed to test B::Generate, which extends B's API. Im uncertain whether this example relys on the extensions.

Also, consider this example:

testop($op2,
   bcons => 'e  <$> const[IV 1] sM ',
   sv => sub { "->NV ".(shift)->NV }, # ??? why NV not IV
   name => 'const',
   flags => &B::OPf_MOD | &B::OPf_WANT_SCALAR);

Here we have an anomaly (bug?) in that the B::Concise rendering says 'const[IV]', but the code that worked (and didnt crash!) was (shift)->NV, not (shift)->IV

ref => 'B::OP'

This is the 1st special %test key; it's not treated as a method, but rather as ok(ref($op), $test{ref}, "isa $op").

class => 'B::OP'

This is TBD, since B::class() fails when called as a method.

bcons => 'string'

This param is truly special - the value is parsed as a B::Concise line, and a set of hypotheses are generated. Each is stored in %tests unless the caller has provided their own. These new tests are then run along with user provided ones.

Currently, at least the following tests are synthesized:

type - this pair is created by calling B::opnumber($tests{name}). This is a round-trip test that is effectively predestined to pass, but which exersizes B code more than otherwize. A few tricks are played here so that nextstate and dbstate are 'equivalent', which prevents false failures when running under debugger.

ref - this is populated based upon a /<.>/ match on the bcons param. The resulting test has little value, since it uses ref($op), not B code. It is however useful to reinforce the 1-many relationship between op-class and op-type.

For nextstate ops, the arg is parsed, and file, line tests are added.

unspecified tests

Since one goal is to *bang* on B as much as possible, we also try various B::*OP methods, protected by $op->can($method). In this example, the last 4 tests are done automatically:

# t  <@> list vKPM/128
ok 159 - list isa B::LISTOP=SCALAR(0xa271a2c)
ok 160 - list->flags is 45
ok 161 - list->name is list
ok 162 - list->type is 142
ok 163 - list->sibling: B::COP=SCALAR(0xa290bac)
ok 164 - list->first: B::OP=SCALAR(0xa290bac)
ok 165 - list->last: B::OP=SCALAR(0xa290bac)
ok 166 - list->children: 3

These tests cannot fail, except by crashing, since no expected result is available. By running them automatically, we tacitly suggest that they be converted to real tests. Specifically, adding children => 3 to %tests will verify that there are really 3 kid ops.

testing siblings, first, last, other

With testop() its possible to test for proper linkage; given a fixed array of ops in exec-order, and knowledge of the ops themselves:

6     <1> entersub[t2] vKS/TARG,1 ->7
-        <1> ex-list K ->6
3           <0> pushmark s ->4
4           <$> const[PV "B"] sM ->5

It should be possible to do something like:

    testop($op[3],
	   bcons    => '<0> pushmark s ->4',
    	   sibling  => $op[4]);

Since pushmark never has kids, its sibling is also its next, and should be testable. CAVEAT - tests like this dont work yet. TBD

test_all_ops(\@ops, $rendering);

This mid-level function takes a B::Concise,-exec rendering, parses it into lines, then calls testop($op, bcons => $line) to test each. This makes it easy to leverage testop's bcons handling.

It is used by test_self_ops(), which adds the ability to get the rendering automatically.

Its weakness is that it hides testop's ability to incorporate customized user tests. An emit-source may be added to improve this.

Current Issues

anomalous IV,NV behavior

If you search the code for (heading), you will find code which attempts to address a discrepancy between how B::Concise renders OP_CONST args (typically \[(IV|NV|PVIV) (\w)+\]) and what testing shows to be fatal.

TODO: Either turn them into exception tests, or examine B::Concise to see how its displaying/determining the type.

Future Development

more parse_bcons() refinements

refactor parse_bcons() op-specific processing, provide plug-on test-subs modelled after anonymous subs probing object returned from '->sv', 1st developed using testop.

follow branches

due to brain-dead op-vector build, we only get ops which are not on a branch, excluding if blocks, for blocks, etc..

Consider a B::Concise callback, or

sentinel

test_self_ops() could also act as a sentinel, whereby only main-code prior to the call is used as B OP cannon-fodder. OTOH, theres no obvious reason why this is useful.