/ 
Pegex-0.73
River stage three • 23 direct dependents • 115 total dependents
/ Pegex::API

The Pegex API

Pegex can be used in many ways: inside scripts, from the command line or as the foundation of a modular parsing framework. This document details the various ways to use Pegex.

At the most abstract level, Pegex works like this:

$result = $parser->new($grammar, $receiver)->parse($input);

Which is to say, abstractly: a Pegex parser, under the direction of a Pegex grammar, parses an input stream, and reports matches to a Pegex receiver, which produces a result.

The parser, grammar, receiver and even the input, are Pegex objects. These 4 objects are involved in every Pegex parse operation, so let's review them briefly:

Pegex::Parser

The Pegex parsing engine. This engine applies the logic of the grammar to an input text. A parser object contains a grammar object and a receiver object. Its primary method is called parse. The default parser engine is non-backtracking, recursive descent. However there are parser subclasses for various alternative types of parsing.

Pegex::Grammar

A Pegex grammar starts as a text file/string composed in the Pegex syntax. Before it can be used in by a Parser it must be compiled. After compilation, it is turned into a data tree consisting of rules and regexes. In modules that are based on a Pegex grammar, the grammar will be compiled into a class file. Pegex itself, uses a Pegex grammar class called Pegex::Pegex::Grammar to parse various Pegex grammars.

Pegex::Receiver

A parser on it's own has no idea what to do with the text it matches. A Pegex receiver is a class that contains methods corresponding to the rules in a grammar. As a rule in the grammar matches, its corresponding receiver method (if one exists) is called with the data that has been matched. It is the receiver's job to take action on the data, often building it into some new structure. Pegex will use Pegex::Tree::Wrap as the default receiver; it produces a reasonably readable tree of the matched/captured data.

Pegex::Input

Pegex abstracts its input streams into an object interface as well. Any operation that can take an input string, can also take an input object. Pegex will turn regular strings into these objects. This is probably the API concept you will encounter the least, but it is covered here for completeness.

All of these object classes can be subclassed to achieve various results. Normally, you will write your own Pegex grammar and a Pegex receiver to achieve a task.

Starting Simple - The pegex Function

The Pegex module exports a function called pegex that you can use for smaller tasks. Here is an example:

use Pegex;
use YAML;

$grammar = "
expr: num PLUS num
num: /( DIGIT+ )/
";

print Dump pegex($grammar)->parse('2+2');

This program would produce:

expr:
- num: 2
- num: 2

Let's review what's happening here. The Pegex module is exporting a pegex function. This function takes a Pegex grammar string as input. Internally this function compiles the grammar string into a grammar object. Then it creates a parser object containing the grammar object and returns it.

The parse method is called on the input string: '2+2'. The string matches, and a nice data structure is returned.

So how was the data structure created? By the receiver object, of course! But we didn't specify one, did we? Nope. It used the default receiver, Pegex::Tree::Wrap. We could have said:

print Dump pegex($grammar, 'Pegex::Tree::Wrap')->parse('2+2');

This receiver basically generates a mapping, where rule names of matches are the keys, and the leaf values are the regex captures.

The more basic receiver called Pegex::Tree generates a tree of sequences that contain just the data (without the rule names). This code:

print Dump pegex($grammar, 'Pegex::Tree')->parse('2+2');

would produce:

- 2
- 2

If we wrote our own receiver class called Calculator like this:

package Calculator;
use base 'Pegex::Tree';

sub got_expr {
    my ($receiver, $data) = @_;
    my ($a, $b) = @$data;
    return $a + $b;
}

Then, this:

print pegex(grammar, 'Calculator')->parse('2+2');

would print:

4

More Explicit Usage

Continuing with the example above, let's see how to do it a little more formally.

use Pegex::Parser;
use Pegex::Grammar;
use Pegex::Tree;
use Pegex::Input;
use YAML;

$grammar_text = "
expr: num PLUS num
num: /( DIGIT+ )/
";

$grammar = Pegex::Grammar->new(text => $grammar_text);
$receiver = Pegex::Tree->new();
$parser = Pegex::Parser->new(
    grammar => $grammar,
    receiver => $receiver,
);
$input = Pegex::Input->new(string => '2+2');

print Dump $parser->parse($input);

This code does the same thing as the first example, but this time we've made all the objects ourselves.

Precompiled Grammars

If you ship a Pegex grammar as part of a CPAN distribution, you'll want it to be precompiled into a module. Pegex makes that easy.

Say the grammar_text about is stored in a file called share/expr.pgx. If you create a module called lib/MyThing/Grammar.pm with content like this:

package MyThing::Grammar;
use base 'Pegex::Grammar';
use constant file => './share/expr.pgx';
sub make_tree {
}
1;

Then run this command line:

perl -Ilib -MMyThing::Grammar=compile

It will rewrite your module to look something like this:

package MyThing::Grammar;
use base 'Pegex::Grammar';
use constant file => './share/expr.pgx';
sub make_tree {
  { '+toprule' => 'expr',
    'PLUS' => { '.rgx' => qr/\G\+/ },
    'expr' => {
      '.all' => [
        { '.ref' => 'num' },
        { '.ref' => 'PLUS' },
        { '.ref' => 'num' }
      ]
    },
    'num' => { '.rgx' => qr/\G([0-9]+)/ }
  }
}
1;

This command found the file where your grammar is, compiled it, and used Data::Dumper to output it back into your module's make_tree method.

This is what a compiled Pegex grammar looks like. As soon as this module is loaded, the grammar is ready to be used by Pegex.

Automatically rebuilding during development with environment variable

If you find yourself needing to compile your grammar module a lot during development, just set this environment variable like so:

export PERL_PEGEX_AUTO_COMPILE=MyThing::Grammar

Now, every time the grammar module is loaded it will check to see if it needs to be recompiled, and do it on the fly.

If you have more than one grammar to recompile, just list all the names separated by commas.

Automatically rebuilding during development using make

Alternatively, if your module uses ExtUtils::MakeMaker, you can have make automatically rebuild your Grammar class if your .pgx file is updated.

Simply add this at the bottom of your Makefile.PL:

sub MY::postamble {
  <<EOF;
lib/MyThing/Grammar.pm : share/expr.pgx
\t\$(PERL) -Ilib -MMyThing::Grammar=compile
EOF
}

See Also