NAME

Asm::Preproc::Lexer - Iterator to split input in tokens

SYNOPSIS

use Asm::Preproc::Lexer;

my @tokens = (
   BLANKS  => qr/\s+/,       sub {()},
   COMMENT => [qr/\/\*/, qr/\*\//],
                             undef,
   QSTR    => [qr/'/],       sub { my($type, $value) = @_;
                                   [$type, 
                                    substr($value, 1, length($value)-2)] },
   QQSTR   => [qr/"/, qr/"/],
   NUM     => qr/\d+/,
   ID      => qr/[a-z]+/,    sub { my($type, $value) = @_; 
                                   [$type, $value] },
   SYM     => qr/(.)/,       sub { [$1, $1] },
);

my $lex = Asm::Preproc::Lexer->new;
$lex->make_lexer(@tokens);

my $lex2 = $lex->clone;

$lex->from(sub {}, @lines);  # read Asm::Preproc::Line from iterator
my $token = $lex->next;      # isa Asm::Preproc::Token
my $token = $lex->();

DESCRIPTION

This module implements a sub-class of Iterator::Simple::Lookahead to read text from iterators and split the text in tokens, according to the specification given to make_lexer constructor.

The objects are Iterator::Simple compatible, i.e. they can be used as an argument to iter().

The tokenizer reads Asm::Preproc::Line objects and splits them in Asm::Preproc::Token objects on each next call. next returns undef on end of input.

FUNCTIONS

new

Creates a new tokenizer object, subclass of Iterator::Simple::Lookahead.

make_lexer must be called to create the tokenizer code before the iterator can be used.

make_lexer

Creates a new tokenizer object for the given token specification. Each token is specified by the following elements:

type

String to identify the token type, unused if the token is discarded (see BLANKS and COMMENT above).

regexp

One of:

1. A single regular expression to match the token at the current input position.

2. A list of one regular expression, to match delimited tokens that use the same delimiter for the start and the end. The token can span multiple lines. See see QSTR above for an example for multi-line single-quoted strings.

3. A list of two regular expressions, to match the start of the token at the current input position, and the end of the token. The token can span multiple lines. See see COMMENT above for an example for multi-line comments.

The regular expression is matched where the previous match finished, and each sub-expression cannot span multiple lines. Parentheses may be used to capture sub-expressions in $1, $2, etc.

It is considered an error, and the tokeninzer dies with an error message when reading input, if some input cannot be recognized by any of the given regexp espressions. Therefore the SYM token above contains the catch-all expression qr/(.)/.

transform (optional)

The optional code reference is a transform subroutine. It receives the type and value of the recognized token, and returns one of:

1. An array ref with two elements [$type, $value], the new type and value to be returned in the Asm::Preproc::Token object.

2. An empty array () to signal that this token shall be dicarded.

As an optimization, the transform subroutine code reference may be set to undef, to signal that the token will be dicarded and there is no use in accumulating it while matching. This is usefull to discard comments upfront, instead of collecting the whole comment, and then pass it to the transform subroutine just to be discarded afterwards. See see COMMENT above for an example of usage.

clone

Creates a copy of this tokenizer object without compiling a new lexing subroutine. The copied object has all pending input cleared.

from

Inserts the given input at the head of the input queue to the tokenizer. The input is either a list of Asm::Preproc::Line objects, or an interator function that returns a Asm::Preproc::Line object on each call.

The input list and interator can also return plain scalar strings, that are converted to Asm::Preproc::Line on the fly, but the information on input file location for error messages will not be available.

The new inserted input is processed before continuing with whatever was already in the queue.

peek

Peek the Nth element from the stream, inherited from Iterator::Simple::Lookahead.

next

Retrieve the next token from the input strean as a Asm::Preproc::Token object, inherited from Iterator::Simple::Lookahead.

AUTHOR, BUGS, SUPPORT, LICENSE, COPYRIGHT

See Asm::Preproc.