NAME

Perl::Lexer - Use Perl5 lexer as a library.

SYNOPSIS

use v5.18.0;
use Perl::Lexer;

my $lexer = Perl::Lexer->new();
my @tokens = @{$lexer->scan_string('1+2')};
for (@tokens) {
    say $_->inspect;
}

Output is:

<Token: THING opval 1>
<Token: ADDOP opnum 63>
<Token: THING opval 2>

DESCRIPTION

THIS LIBRARY IS WRITTEN FOR RESEARCHING PERL5 LEXER API. THIS MODULE USES PERL5 INTERNAL API. DO NOT USE THIS.

Perl::Lexer is a really hackish library for using Perl5 lexer as a library.

MOTIVATION

The programming language provides lexer library for itself is pretty nice. I want to tokenize perl5 code by perl5.

Of course, this module uses Perl5's private APIs. I hope these APIs turn into public.

If we can use lexer, we can write a source code analysis related things like Test::MinimumVersion, and other things.

WHAT API IS NEEDED FOR WRITING MODULES LIKE THIS.

Open the token informations for XS hackers.

Now, token name, type, and arguments informations are hidden at toke.c and perly.h.

I need to define `PERL_CORE` for use it... It's too bad.

And, I take token_type and debug_tokens from toke.c.

METHODS

my $lexer = Perl::Lexer->new();

Create new Perl::Lexer object.

$lexer->scan_string($code: Str) : ArrayRef[Str]

Tokenize perl5 code. This method returns arrayref of Perl::Lexer::Token.

LICENSE

Copyright (C) Tokuhiro Matsuno.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

AUTHOR

Tokuhiro Matsuno <tokuhirom@gmail.com>