/ 
Outthentic-0.0.23
River stage zero No dependents
/ Outthentic

Name

Outthentic

Synopsis

Generic testing, reporting, monitoring framework consuming Outthentic::DSL.

Install

$ cpanm Outthentic

Short introduction

This is a quick tutorial on outthentic usage.

Story being tested

Story is just a perl script that yields something into stdout:

$ cat story.pl

print "I am OK\n";
print "I am outthentic\n";

Sometimes we can also call story file as scenario.

Check file

Story check is a bunch of lines stdout should match. Here we require to have `I am OK' and `I am outthentic' lines in stdout:

$ cat story.check

I am OK
I am outthentic

Story run

Story run is process of verification of your story. A story verification is based on rules defined in story check file.

The verification process consists of:

  • executing story file and saving stdout into file.

  • validating stdout against a story check.

  • returning result as the list of statuses, where every status relates to a single rule.

See also story runner.

Suite

A bunch of related stories is called project or suite. Sure you may have more then one story at your project. Just create a new directories with story files inside:

$ mkdir hello
$ echo 'print "hello"' > hello/story.pl
$ echo hello > hello/story.check

Now run the suite with strun command:

$ strun
ok 1 - perl /home/vagrant/projects/outthentic/examples/hello/story.pl succeeded
ok 2 - stdout saved to /tmp/.outthentic/29566/QKDi3p573L
ok 3 - output match 'hello'
1..3
ok
/tmp/.outthentic/29566/home/vagrant/projects/outthentic/examples/hello/world/story.t ..
ok 1 - perl /home/vagrant/projects/outthentic/examples/hello/world/story.pl succeeded
ok 2 - stdout saved to /tmp/.outthentic/29566/xC3wrsS195
ok 3 - output match 'I am OK'
ok 4 - output match 'I am outthentic'
1..4
ok
All tests successful.
Files=2, Tests=7,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.09 cusr  0.01 csys =  0.13 CPU)
Result: PASS

Calculator project example

Here is more detailed tutorial where we will build a test suite for calculator program.

Let's repeat it again - there are three basic outthentic entities:

  • project ( suite )

  • story files ( scenarios )

  • story checks ( rules )

Project

Outthentic project is a bunch of related stories. Every project is represented by a directory.

Let's create a project to test a simple calculator application:

$ mkdir calc-app
$ cd calc-app

Stories

Stories are just perl scripts placed at project sub-directories and named story.pl.

Every story is a small program with stdout gets tested.

Let's create two stories for our calc project. One story for `addition' operation and another for `multiplication':

# story directories

$ mkdir addition # a+b
$ mkdir multiplication # a*b


# story files

$ cat  addition/story.pl
use MyCalc;
my $calc = MyCalc->new();
print $calc->add(2,2), "\n";
print $calc->add(3,3), "\n";

$ cat  multiplication/story.pl
use MyCalc;
my $calc = MyCalc->new();
print $calc->mult(2,3), "\n";
print $calc->mult(3,4), "\n";

Story check files

Story checks file contain validation rules for story.pl files. Every story.pl is always accompanied by story.check file. Story check files should be placed at the same directory as story.pl file.

Lets add some rules for multiplication and addition stories:

$ cat addition/story.check
4
6
 
$ cat multiplication/story.check
6
12

And finally lets run test suite:

$ strun

Story term ambiguity

Sometimes when we speak about stories we mean an elementary scenario executed by story runner and represented by a couple of files - story.pl,story.check. In other cases we mean just a story.pl file or even story.check given separately. The one should always take the context into account when talking about stories to avoid ambiguity.

Story runner

Story runner - is a script to run outthentic stories. It is called strun.

Runner consequentially goes several phases:

A compilation phase.

Stories are converted into perl test files *.t ( compilation phase ) and saved into temporary directory.

An execution phase.

Prove utility recursively executes test files under temporary directory and thus gives a final suite execution status.

So after all outthentic project is just perl test project with *.t files inside, the difference is that while with common test project *.t files are created by user, in outthentic project *.t files are generated by story files.

Story checks syntax

Outthentic consumes Outthentic DSL, so story checks are just rules defined in terms of Outthentic DSL - a language to validate unstructured text data.

A few ( not all ) usage examples listed below.

  • plain strings checks

Often all you need is to ensure that stdout has some strings in:

# stdout
HELLO
HELLO WORLD
123456


# check list
HELLO
123

# validation output
OK - output matches 'HELLO'
OK - output matches 'HELLO WORLD'
OK - output matches '123'

  • regular expressions

You may use regular expressions as well:

# check list
regexp: L+
regexp: \d


# validation output
OK - output matches /L+/
OK - output matches /\d/

See check-expressions in Outthentic::DSL documentation pages.

  • generators

Yes you may generate new check entries on run time:

# original check list
   
Say
HELLO
   
# this generator creates 3 new check expressions:
   
generator: [ qw{ say hello again } ]
   
# final check list:
   
Say
HELLO
say
hello
again

See generators in Outthentic::DSL documentation pages.

  • inline perl code

What about inline arbitrary perl code? Well, it's easy!

# check list
regexp: number: (\d+)
validator: [ ( capture()->[0] '>=' 0 ), 'got none zero number') ];

See perl expressions in Outthentic::DSL documentation pages.

  • text blocks

Need to validate that some lines goes successively?

    # stdout

    this string followed by
    that string followed by
    another one string
    with that string
    at the very end.


    # check list
    # this text block
    # consists of 5 strings
    # goes consequentially
    # line by line:

    begin:
        # plain strings
        this string followed by
        that string followed by
        another one
        # regexps patterns:
    regexp: with (this|that)
        # and the last one in a block
        at the very end
    end:

See comments-blank-lines-and-text-blocks in Outthentic::DSL documentation pages.

Hooks

Story hooks are extension points to change story run process.

It's just files with perl code gets executed in the beginning of a story.

You should name your hooks as story.pm and place them into story directory:

$ cat addition/story.pm
diag "hello, I am addition story hook";
sub is_number { [ 'regexp: ^\\d+$' ] }
 

$ cat addition/story.check
generator: is_number

Reasons why you might need a hooks:

  • redefine story stdout

  • define generators

  • call downstream stories

  • other custom code

Hooks API

Story hooks API provides several functions to hack into story run process:

Redefine stdout

set_stdout(string)

Using set_stdout means that you never execute a story.pl to get a stdout, but instead you set stdout on your own side.

This might be helpful when for some reasons you can't produce a stdout via story.pl file:

This is simple an example :

$ cat story.pm
set_stdout("THIS IS I FAKE RESPONSE\n HELLO WORLD");

$ cat story.check
THIS IS FAKE RESPONSE
HELLO WORLD

You may call set_stdout() more then once:

set_stdout("HELLO WORLD");
set_stdout("HELLO WORLD2");

A final stdout will be:

HELLO WORLD
HELLO WORLD2

Upstream and downstream stories

It is possible to run one story from another with the help of downstream stories.

Downstream stories are reusable stories or modules.

Story runner never executes downstream stories directly, instead of downstream story always gets called from the upstream one:

$ cat modules/create_calc_object/story.pm
# this is a downstream story
# to make story downstream
# simply create story 
# in modules/ directory
use MyCalc;
our $calc = MyCalc->new();
set_stdout(ref($calc));
 
$ cat modules/create_calc_object/story.check
MyCalc

 
$ cat addition/story.pm
# this is a upstream story
# to run downstream story

# call run_story function
# inside upstream story hook
# with a single parameter - story path,
# note that you don't have to
# leave modules/ directory in the path

run_story( 'create_calc_object' );

# here $calc object is created by 
# create_calc_object story
# so we can use it!

our $calc->addition(2,2);

Here are the brief comments to the example above:

  • to make story as downstream simply create story at modules/ directory

  • call run_story(story_path) function inside upstream story hook to run downstream story.

  • you can call as many downstream stories as you wish.

  • you can call the same downstream story more than once.

Here is an example code snippet:

$ cat story.pm
run_story( 'some_story' )
run_story( 'yet_another_story' )
run_story( 'some_story' )

  • stories variables

You may pass variables to downstream story with the second argument of run_story() function:

run_story( 'create_calc_object', { use_floats => 1, use_complex_numbers => 1, foo => 'bar'   }  )

Story variables get accessed by story_var() function:

$ cat create_calc_object/story.pm
story_var('use_float');
story_var('use_complex_numbers');
story_var('foo');

  • downstream stories may invoke other downstream stories

  • you can't use story variables in a none downstream story

One word about sharing state between upstream/downstream stories.

As downstream stories get executed in the same process as upstream one there is no magic about sharing data between upstream and downstream stories.

The straightforward way to share state is to use global variables:

# upstream story hook:
our $state = [ 'this is upstream story' ]

# downstream story hook:
push our @$state, 'I was here'

Story variables accessors

There are some useful variables exposed by hooks API:

  • project_root_dir() - Root directory of outthentic project.

  • test_root_dir() - Test root directory. Root directory of generated perl test files , see also story runner.

  • config() - Returns suite configuration hash object. See also suite configuration.

  • host() - Returns value of `--host' parameter.

Ignore unsuccessful codes when run stories

Every story is a perl script gets run by perl system() function returning an exit code.

None zero exit codes result in test failures, this default behavior, to disable this say in hook file:

$ cat story.pm
ignore_story_err(1);

PERL5LIB

$project_root_directory/lib path gets added to $PERL5LIB variable.

This make it easy to place custom modules under project root directory:

$ cat my-app/lib/Foo/Bar/Baz.pm
package Foo::Bar::Baz;
...

$ cat  hook.pm
use Foo::Bar::Baz;
...

Story runner client

$ strun <options>

Options

  • --root

Root directory of outthentic project. If root parameter is not set current working directory is assumed as project root directory.

  • --debug

Enable/disable debug mode:

* Increasing debug value results in more low level information appeared at output

* Default value is 0, which means no debugging 

* Possible values: 0,1,2,3

  • --match_l

Truncate matching strings. In a TAP output truncate matching strings to {match_l} bytes; default value is 200.

  • --story

Run only single story. This should be file path without extensions ( .pl, .check ):

foo/story.pl
foo/bar/story.pl
bar/story.pl

--story 'foo' # runs foo/ stories
--story foo/story # runs foo/story.pl
--story foo/bar/ # runs foo/bar/ stories

  • --prove

Prove parameters. See prove settings section for details.

  • --host

This optional parameter sets base url or hostname of a service or application being tested.

  • --ini

Configuration ini file path.

See suite configuration section for details.

  • --yaml

Yaml configuration file path.

See suite configuration section for details.

Suite configuration

Outthentic projects are configurable. Configuration data is passed via configuration files.

There are two type of configuration files are supported:

  • .Ini style format

  • YAML format

.Ini style configuration files are passed by --ini parameter

$ strun --ini /etc/suites/foo.ini

$ cat /etc/suites/foo.ini

[main]

foo = 1
bar = 2

There is no special magic behind ini files, except this should be Config Tiny compliant configuration file.

Or you can choose YAML format for suite configuration by using --yaml parameter:

$ strun --yaml /etc/suites/foo.yaml

$ cat /etc/suites/foo.yaml

main:
  foo : 1
  bar : 2

Unless user sets path to configuration file explicitly by --ini or --yaml story runner looks for the files named suite.ini and then ( if suite.ini is not found ) for suite.yaml at the current working directory.

If configuration file is passed and read a related configuration data is accessible via config() function, for example in story hook file:

$ cat story.pm

my $foo = config()->{main}->{foo};
my $bar = config()->{main}->{bar};

Runtime configuration

WARNING: this feature is quite experimental, needs to be tested and is could be buggy, don't use it unless this warning will be removed

Runtime configuration parameters is way to override suite configuration data. Consider this example:

$ cat suite.ini
[foo]
bar = 10
  
  
$ strun --param foo.bar=20

This way we will override foo.bar to value `20'.

It is possible to override any data in configuration files, for example arrays values:

$ cat suite.ini

[foo]
bar = 1
bar = 2
bar = 3


$ suite --param foo.bar=11 --param foo.bar=22 --param foo.bar=33

TAP

Story runner emit results in a TAP format.

You may use your favorite TAP parser to port result to another test / reporting systems.

Follow TAP documentation to get more on this.

Here is example for having output in JUNIT format:

strun --prove "--formatter TAP::Formatter::JUnit"

Prove settings

Story runner uses prove utility to execute generated perl tests, you may pass prove related parameters using --prove-opts. Here are some examples:

strun --prove "-Q" # don't show anythings unless test summary
strun --prove "-q -s" # run prove tests in random and quite mode

Environment variables

  • match_l - In a suite runner output truncate matching strings to {match_l} bytes. See also --match_l in options.

  • outth_show_story - If set, then content of story.pl file gets dumped in TAP output.

Examples

An example outthentic project lives at examples/ directory, to run it say this:

$ strun --root examples/

AUTHOR

Aleksei Melezhik

Home Page

https://github.com/melezhik/outthentic

See also

  • Sparrow - outthentic suites manager.

  • Outthentic::DSL - Outthentic::DSL specification.

  • Swat - web testing framework consuming Outthentic::DSL.

  • Perl prove, TAP, Test::More

Thanks

To God as the One Who inspires me to do my job!