/ 
Syntax-Highlight-Basic-0.1.1
/ Syntax::Highlight::Basic::Parser

NAME

Syntax::Highlight::Basic::Parser - Tokenize source code for syntax highlighting

SYNOPSIS

use Syntax::Highlight::Basic::Parser;

my $parser = Syntax::Highlight::Basic::Parser->new(
    language    => 'perl',
    syntax_dirs => ['/path/to/custom/syntax'],
);

my $tokens = $parser->parse("if (\$x) {\n    print \"hello\";\n}");

# $tokens is an arrayref of arrayrefs of token hashes:
# [
#     [ { class => 'Statement', sub_group => 'Conditional', text => 'if' },
#       { class => 'whitespace', sub_group => undef, text => ' ' },
#       ... ],
#     [ ... ],
# ]

DESCRIPTION

Syntax::Highlight::Basic::Parser reads .shb syntax data files and tokenizes source code into an array of arrays of token hashes. Each token contains the matched text, its Vim highlight group classification, and an optional sub-group.

The parser supports:

  • Keyword matching (exact word matches with word boundaries)

  • Regex pattern matching (single-line patterns)

  • Multi-line region matching (strings, block comments)

  • Fallback tokenizer for unrecognized languages

CONSTRUCTOR

new(%options)

Creates a new Parser instance.

Options:

language

The programming language name (e.g., 'perl', 'python'). The parser searches for a corresponding .shb file. Pygments-compatible aliases are resolved via the built-in mapping table.

syntax_dirs

An array reference of additional directories to search for .shb files. User-supplied directories are searched before the module's built-in share/syntax/ directory.

METHODS

parse($code)

Tokenizes the given source code string.

Returns an array reference of array references. Each inner array corresponds to one line of input and contains token hash references:

{ class => 'Statement', sub_group => 'Keyword', text => 'if' }

Special class values:

'whitespace' — spaces and tabs
'text' — unrecognized text

SYNTAX FILES

See docs/syntax-format.md for the complete .shb file format reference.

VERSION

0.1.0

AUTHOR

Sandor Patocs

LICENSE

This module is licensed under the same terms as Perl itself.

SYNTAX FILES

Syntax definitions are stored in .shb files. The module ships with definitions for many languages in its share/syntax/ directory. Users can add custom definitions by creating their own .shb files and passing the directory via the syntax_dirs option.

For a complete reference on the .shb file format, see "syntax-format.md" in docs in the distribution.

SEE ALSO

Syntax::Highlight::Basic, Syntax::Highlight::Basic::Output::Pygments, Syntax::Highlight::Basic::Output::HTML, Syntax::Highlight::Basic::Output::Ansi

1 POD Error

The following errors were encountered while parsing the POD:

Around line 708:

Non-ASCII character seen before =encoding in '—'. Assuming UTF-8