NAME

Marpa::UrHTML - High-level HTML Parser

SYNOPSIS

Delete all tables:

use Marpa::UrHTML qw(urhtml);

my $with_table = 'Text<table><tr><td>I am a cell</table> More Text';
my $no_table   = urhtml( \$with_table, { table => sub { return q{} } });

Delete everything but tables:

my %handlers_to_keep_only_tables = (
    table  => sub { return Marpa::UrHTML::original() },
    ':TOP' => sub { return \( join q{}, @{ Marpa::UrHTML::values() } ) }
);
my $only_table = urhtml( \$with_table, \%handlers_to_keep_only_tables );

The above works by turning the original text of the HTML into values and concatenating the values at the top of the parse. The same logic works even if a table is very defective:

my $with_bad_table = 'Text<tr>I am a cell</table> More Text';
my $only_bad_table =
    urhtml( \$with_bad_table, \%handlers_to_keep_only_tables );

Delete all comments:

my $with_comment = 'Text <!-- I am a comment --> I am not a comment';
my $no_comment   = urhtml( \$with_comment,
    { ':COMMENT' => sub { return q{} } });

By default text is passed through unchanged, so that the user need only specify semantic actions for those components she wants changed. To change the title of a document:

my $old_title = '<title>Old Title</title>A little html text';
my $new_title = urhtml(
    \$old_title,
    {   'title' => sub { return '<title>New Title</title>' }
    }
);

Delete all elements with a class attribute of "delete_me":

my $stuff_to_be_edited = '<p>A<p class="delete_me">B<p>C';
my $edited_stuff       = urhtml( \$stuff_to_be_edited,
    { '.delete_me' => sub { return q{} } });

Marpa::UrHTML recognizes elements even if they have missing start and/or end tags. Marpa::UrHTML can supply missing tags:

sub supply_missing_tags {
    my $tagname = Marpa::UrHTML::tagname();
    return if $empty_elements{$tagname};
    return ( Marpa::UrHTML::start_tag() // "<$tagname>\n" )
        . Marpa::UrHTML::contents() .
        ( Marpa::UrHTML::end_tag() // "</$tagname>\n" );
}
my $html_with_just_a_title = '<title>I am a title and That is IT!';
my $valid_html_with_all_tags =
    urhtml( \$html_with_just_a_title, { q{*} => \&supply_missing_tags } );

Marpa::UrHTML understands the hierarchical structure of an HTML document. Finding the maximum nesting depth in elements is straightforward:

sub depth_below_me {
    return List::Util::max( 0, @{ Marpa::UrHTML::values() } );
}
my %handlers_to_calculate_maximum_element_depth = (
    q{*}   => sub { return 1 + depth_below_me() },
    ':TOP' => sub { return depth_below_me() },
);
my $maximum_depth_with_just_a_title = urhtml( \$html_with_just_a_title,
    \%handlers_to_calculate_maximum_element_depth );

Marpa::UrHTML tracks actual elements, however tagged. The above code returns the same depth for $valid_html_with_all_tags as for $html_with_just_a_title.

DESCRIPTION

Marpa::UrHTML does "high-level" parsing of HTML. It allows handlers to be specified for elements, terminals and other components in the hierarchical structure of an HTML document. Marpa::UrHTML is an extremely liberal HTML parser. Marpa::UrHTML does not reject any documents, no mater how poorly they fit the HTML standards.

THE `urhtml` STATIC METHOD

The interface to Marpa::UrHTML is through the Marpa::UrHTML::urhtml static method. It is the only Marpa::UrHTML method not part of the API for the semantic actions.

urhtml takes one or more arguments. The first argument is required, and must be a pointer to a string to be parsed as HTML. The second and subsequent arguments (all optional) are hash references with handler descriptions. (See the synopsis for several examples of calls using the urhtml method.)

CSS-style Handler Options

Handler descriptions in Marpa::UrHTML are key-value pairs in a hash. In each pair, the key is a CSS-style handler specifier, and the value is a closure, which is called the action for the handler.

Specifiers are "CSS-style" -- their syntax imitates some of the basic cases of CSS specifiers. No attempt is planned to implement the full CSS specifier syntax.

Supported specifier syntaxes are as follows:

Tagname Specifiers

table  => sub { return Marpa::UrHTML::original() },

If a specifier contains no special characters it is taken as the name of an element. (A "special" character is anything except an alphanumeric, a hyphen or an underscore.) Consistent with HTML::Parser's default behavior, element names must be specified in lowercase.

Class Specifiers

A specifier which begins with a dot or period (such as ".delete_me") will match any element whose class attribute is "delete_me".

Tagname-Class Pair Specifiers

A specifier which contains a dot or period somewhere other than the first position (such as "span.label") is treated as a dotted tagname-class pair. Its action will be called for any component whose tagname and class attribute both match the specifiers.

The Tagname Wildcard Specifier

A specifier of just an asterisk ("*") matches all elements. Be careful to note that matching all elements is not the same as matching all components. The element wildcard specifier will not match any pseudoclasses.

Pseudoclass Specifiers

':COMMENT' => \&delete_it

A specifier which begins with a colon (":") is a pseudoclass. Marpa::UrHTML defines pseudoclasses to deal with terminals and other non-element components of the HTML hierarchy.

Conflicting Specifiers

Only one handler is called for each component. Where an element component matches several specifiers, the action is picked based on the most specific match.

1. Matches by tagname-class pair are the most specific.
2. Matches by class are the next most specific.
3. Matches by tagname are considered less specific than matches by class.
4. The wildcard match is the least specific.

Here's an example:

my $html = <<'END_OF_HTML';
<span class="high">High Span</span>
<span class="low">Low Span</span>
<div class="high">High Div</div>
<div class="low">Low Div</div>
<div class="oddball">Oddball Div</div>
END_OF_HTML

my $result = Marpa::UrHTML::urhtml(
    \$html,
    {   q{*} => sub {
            return 'wildcard handler: ' . Marpa::UrHTML::contents();
        },
        'head' => sub { return Marpa::UrHTML::literal() },
        'html' => sub { return Marpa::UrHTML::literal() },
        'body' => sub { return Marpa::UrHTML::literal() },
        'div'  => sub {
            return '"div" handler: ' . Marpa::UrHTML::contents();
        },
        '.high' => sub {
            return '".high" handler: ' . Marpa::UrHTML::contents();
        },
        'div.high' => sub {
            return '"div.high" handler: ' . Marpa::UrHTML::contents();
        },
        '.oddball' => sub {
            return '".oddball" handler: ' . Marpa::UrHTML::contents();
        },
    }
);

Here is what $result would contain after the above code was run:

".high" handler: High Span
wildcard handler: Low Span
"div.high" handler: High Div
"div" handler: Low Div
".oddball" handler: Oddball Div

Details of the Specifier Syntax

For elements and class names only alphanumerics, hyphens and underscores are supported. Elements must be specified in lowercase, but they will match tagnames in the original document on a case-insensitive basis.

Forcing element names to be lowercase follows the default behavior of HTML::Parser, which coerces all tagnames to lowercase. This is consistent with the HTML standards. It is not consistent with the XML standards, and an option to configure this behavior may be added in the future.

Pseudoclass names special to Marpa::UrHTML are case-sensitive, and must be all uppercase. Lowercase is reserved for CSS pseudoclasses. The CSS standard specifies that its pseudoclass names are case-indifferent. No CSS pseudoclasses are supported at this writing.

PSEUDOCLASSES

Marpa::UrHTML uses HTML::Parser to do its low-level parsing. HTML::Parser "events" become the terminals for Marpa::UrHTML's parser.

Besides terminals and elements, three other HTML components are recognized: the physical document's SGML prolog (:PROLOG), the physical document's SGML trailer (:TRAILER), and the HTML document as a whole (:TOP).

:CDATA

The :CDATA pseudoclass specifies the action for CDATA terminals. Its action is called once for each non-whitespace raw text event. (Raw text is text in which any markup and entities should be left as is.)

More precisely, a :CDATA terminal is created from any HTML::Parser text event that has the is_cdata flag on, and that contains a non-whitespace character as defined in the HTML 4.01 specification (http://www.w3.org/TR/html4/struct/text.html#h-9.1).

:COMMENT

The :COMMENT pseudoclass specifies the action for HTML comments. Its action is called once for every HTML::Parser comment event that is not reclassed as cruft.

:CRUFT

The :COMMENT pseudoclass specifies the action for cruft. Its action is called once for every HTML::Parser event that Marpa::UrHTML reclasses as cruft.

Marpa::UrHTML reclasses terminals as cruft when they do not fit the structure of an HTML document. One example of a terminal that Marpa::UrHTML would reclass as cruft is a </head> end tag in the HTML body.

Reclassing terminals as cruft is only done as the last resort. When it can, HTML::Parser forgives violations of the HTML standards and accepts terminals as non-cruft.

Cruft is treated in much the same way as comments. It is preserved, untouched, in the original text view.

:DECL

The :DECL pseudoclass specifies the action for SGML declarations. Its action is called once for every HTML::Parser declaration event that is not reclassed as cruft.

:PCDATA

The :PCDATA pseudoclass specifies the action for PCDATA terminals. Its action is called once for each non-whitespace non-raw text event.

More precisely, a :PCDATA terminal is created from any HTML::Parser text event that has the is_cdata flag off, and that contains a non-whitespace character as defined in the HTML 4.01 specification (http://www.w3.org/TR/html4/struct/text.html#h-9.1).

Markup and entities in :PCDATA text are expected to be interpreted eventually, but it can be counter-productive to do this during parsing. An application may, for example, be rewriting a document for display on the web. In that case it will often need to leave markup and entities for the client's browser to interpret.

To allow flexibility, Marpa::UrHTML leaves interpretation of markup and entities entirely to the user. A user who does choose to do the interpretation may do it in the actions, or deal with it in post-processing. CPAN has excellent tools for this, some of which are part of HTML::Parser.

:PI

The :PI pseudoclass specifies the action for SGML processing instructions. Its action is called once for every HTML::Parser process event that is not reclassed as cruft.

:PROLOG

The :PROLOG pseudoclass specifies the action for SGML prolog. This is the part of the HTML document which precedes the HTML root element. Components valid in the prolog include SGML comments, processing instructions and whitespace.

:TOP

The action specified for the :TOP pseudoclass will be called once and only once in every parse, and will be the last action called in every parse. The :TOP component is the entire physical document, including the SGML prolog, the root element, and the SGML trailer. All the other HTML components in a document will be descendants of the :TOP component.

The :TOP action is unique, in that there is always an action for it, even if one is not specified. The urhtml method returns the value returned by the :TOP action. The default :TOP action returns a reference to a string with the literal text value of all of its descendants.

:TRAILER

The :TRAILER pseudoclass specifies the action for SGML trailer. This is the part of the HTML document which precedes the HTML root element. Components valid in the prolog include SGML comments, processing instructions, and whitespace. Cruft can also be found here, though for Marpa::UrHTML that is a last resort.

:WHITESPACE

A Marpa::UrHTML :WHITESPACE terminal is created for every HTML::Parser text event that is entirely whitespace as defined in the HTML 4.01 specification (http://www.w3.org/TR/html4/struct/text.html#h-9.1). Whitespace is acceptable in places where non-whitespace is not, and the difference can be very significant structurally.

VIEWS

I hope the synopsis convinces the reader that the action semantics of Marpa::UrHTML are natural. This naturalness is achieved at the price of some novelty. This section explains the ideas behind the action API. Depending on taste, readers may want to skip this section and go straight to the API.

The components of an HTML document form a hierarchy, with the TOP component on top, and the terminals on the bottom. The traditional syntax tree method requires semantic actions to know precisely what children every component will have. This processing model is not a good fit to HTML. Marpa::UrHTML gives the writer of semantic actions "views" of each component that better fit situations where the number and type of children is unknown or vaguely defined.

Marpa::UrHTML's action semantics therefore often focus more on a components descendants than its direct children. The terms ancestor and descendant are used in the standard way: A component is an ancestor of a second component if it is above that second component in the hierarchy. In this case, also, the second component is a descendant of the first one.

The Original View

The original view sees the text of a component as it was originally passed to the parser. The original view never changes. In its "pure form" call, the original view is seen through the Marpa::UrHTML::original API call.

The Terminal View

The terminal view sees the terminals for a component. The terminal view sees the terminals corresponding to the original text of a component. The terminal view never changes. The terminal view is usually seen as part of other views.

At this writing the API does not contain a "pure form" terminal view call. For a terminal view of the whole HTML document, the HTML::PullParser does the job with significantly lower overhead. For a terminal view of the sections with no values defined, the descendants view (described below is equivalent to the terminals view).

The Values View

When actions are called, they return a value. If that value is defined, it becomes visible to the values view of its ancestors. The values view of a component sees the visible values for its descendants.

The values view is an array, with the values ordered according to the lexical order of the components whose actions returned them. If no descendants have visible values, then the values view is a zero-length array.

The values view is hierarchical. When a component produces a visible value, it makes the values of its descendants disappear. That is, whenever the semantic action for a component X returns anything other than a Perl undef, it has two effects:

That return value becomes the visible value associated with component X.
All the values previously visible due to semantic actions for the descendants of component X disappear.

Values which disappear are gone forever. There is no mechanism to make them "reappear".

As a special case, if an action for a component returns a Perl undef, not only do the values of all its descendants disappear, the component for the action also will not appear in the values view. When its semantic action returns undef, a component permanently "drops out" of the values view taking all descendants with it.

In its "pure form", the original view is seen through the Marpa::UrHTML::values API call.

The Literal View

The literal view can be thought of as a mix between the original view and the values view. It sees a text string, like the original view. But unlike the original view, the literal view includes the visible values.

Values appear in the literal view in stringized form. For sections of the original text without visible values, the literal view is the same as the original view. In all Marpa::UrHTML's views, whether descendants are seen as text or values, they are seen in the original lexical order. In its "pure form", the literal view is seen through the Marpa::UrHTML::literal API call.

The Descendants View

Just as the literal view can be thought of as a mix between the original view and the values view, the descendants view can be thought of a mix between the terminal view and the values view.

The descendants view sees an array of element with data for each of the component's descendants, in lexical order. Where a value is visible, the descendants view sees data for the component with the visible value. Where no value is visible, the descendants view sees data for the terminals. This means that when no values are visible, the descendants view is the same as the terminals view.

The descendants view is implemented via the "Marpa::UrHTML::descendants" method. It is the most fine-grained and detailed way to look at the descendants of a component. The descendants view can do anything that the other views can do, but the other views should be prefered when they fit the application. The other views are typically more intuitive and efficient.

Views versus Syntax Trees

Views are a generalization of the traditional method for processing semantics: syntax trees. The values view is the view that most closely resembles a syntax tree. But there are important differences.

The syntax tree model, in its pure form, required the semantic actions to define exactly how many and what kind of immediate children each node had. Each node in a syntax tree working with its immediate children. And children in a syntax tree always appeared as values.

The values view, on the other hand, sees all descendants, but only those which make themselves visible. The values view allows pieces of the tree to decide when they will come into sight and when they will fall out of view. Because of this, the values view lends itself to being mixed with other views.

Views and Efficiency

In most applications, views are more efficient than syntax trees. From the Marpa::UrHTML point of view, Tradtional syntax tree processing, in its pure form, corresponds to the case in which all components have actions which return defined values. For Marpa::UrHTML this is close to its worst case.

Marpa::UrHTML optimizes for unvalued components. Unvalued components are represented as terminal spans. Adjacent descendant spans are automatically merged. This means the size and time required do not increase as processing rises up the component hierarchy.

Terminal views are calculated on a just-in-time basis when they are requested through the action API. The terminal view is produced quickly from the merged terminal span.

Original views are also calculated on a just-in-time basis as requested. Each terminal tracks the text it represents as a character span. The original text can be quickly reconstructed as the text from the first character location of its component's first terminal to the last character location of the component's last terminal.

When a handler does not need to return a value, the most efficient thing to do is to return undef. This reverts that component and all its descendants to the efficient unvalued representation.

THE SEMANTIC ACTION API

Marpa::UrHTML's semantic action API is implemented mainly through context-aware static methods. No arguments are passed to the user's semantics action callbacks. Instead the semantic actions get whatever data they need by calling these static methods.

API Static Methods

Marpa::UrHTML::attributes

Returns a hash ref to the attributes of the start tag. This hash ref is exactly the hash ref returned for the attr arg specification of HTML::Parser. The attributes API method returns an empty hash if there were no attributes, if there was no start tag for this element, or if the current component is not an element.

Marpa::UrHTML::contents

For an element, returns the literal view of the contents. The contents of an element are its entire text except for its start tag and its end tag. For an non-element component, returns undef.

Marpa::UrHTML::descendants

This static method implements the descendants view. It takes one argument, the "dataspec". The dataspec is a string specifying the data to be returned for each descendant. The descendants method returns a reference to an array with one element per descendant, in lexical order. Each element in the array is a reference to an array whose elements are the per-descendant data requested in the string.

The descendant data specification string has a syntax similar to that of the argspec strings of HTML::Parser. Details of that syntax are given below

Marpa::UrHTML::values

The Marpa::UrHTML::values method implements the values view. It returns a pointer to an array of the descendant values visible from this component, in lexical order. No elements of this array will be undefined. The array will be zero length if no descendant has a visible value.

Marpa::UrHTML::end_tag

For an element with an explicit end tag, returns the original text of the end tag. For non-element components, returns undef. For elements with no end tag, returns undef.

Marpa::UrHTML::literal

The Marpa::UrHTML::literal method implements the literal view. Returns a string containing the literal view of the component -- its text as modified by any the visible values of its descendants.

Marpa::UrHTML::literal_ref

Returns a reference to a string containing the literal view of the component. This can be useful for very long strings.

Marpa::UrHTML::offset

Returns the start offset of the component. This is a zero-based location in the original text string.

Marpa::UrHTML::original

The Marpa::UrHTML::original method implements the original view. Returns a string containing the original view of the component -- its text as modified by any the visible values of its descendants.

Marpa::UrHTML::start_tag

For an element with an explicit start tag, returns the original text of the start tag. For non-element components, returns undef. For elements with no start tag, returns undef.

Marpa::UrHTML::tagname

For an element component returns its tagname. There is a tagname even if there are no explicit tags. It is determined based on structure, For non-element components, returns undef.

Marpa::UrHTML::title

Returns the values of the title attribute. For a non-element component, returns undef. If there was explicit start tag, returns undef. If there was no title attribute, returns undef.

Dataspecs

Marpa::UrHTML::descendants('token_type,literal,element')

The data specification string, or dataspec, is a comma separated list of descendant data specifiers for per-descendant data. The Marpa::UrHTML::descendants method takes a dataspec as its argument. The Marpa::UrHTML::descendants method returns a reference to an array of references to arrays of per-descendant data. The contents of the per-descendant data arrays and their order is as specified by the dataspec. These are the valid descendant data specifiers:

element: For an element descendant, returns the tagname. The tagname is known even if there were not explicit tags. For non-element descendants, returns undef.
literal: Returns a string containing the literal view of the descendant.
original: Returns a string containing the original view of the descendant.
pseudoclass: If the descendant is a pseudoclass terminal, returns the name of the pseudoclass. In all other cases, returns undef.
token_type: If the descendant is a terminal, returns the token type. These token types are the event types from HTML::Parser: "T" for text, "S" for a start tag, "E" for an end tag, "PI" for a processing instruction, "D" for an SGML desclaration, and "C" for a comment, For elements with visible values, returns undef.
value: For element descendants with a value, returns that value. In all other cases, returns undef.

The Instance Hash

Each Marpa::UrHTML instance makes available a per-instance variable as a scratchpad for the application: $Marpa::UrHTML::INSTANCE. Each call to Marpa::UrHTML::urhtml internally creates a $Marpa::UrHTML::INSTANCE variable which is reserved for that application using the local keyword. Marpa::UrHTML::urhtml initializes it to an empty hash, but after that does not touch it. When programming via side effects is more natural than passing data up the parse tree (and it often is), $Marpa::UrHTML::INSTANCE can be used to store the data.

Ordinarily, $Marpa::UrHTML::INSTANCE is destroyed, with the rest of the parse instance, when Marpa::UrHTML::urthml returns. But it can be useful for the :TOP semantic action to return $Marpa::UrHTML::INSTANCE as the value of the parse.

Undefined Actions versus Actions Which Return `undef`

It is worth emphasizing that the effect of not defining a semantic action for a component is different from the effect of defining a semantic action which returns a Perl undef. The difference lies in what happens to any visible values of the descendants of that component.

Where no action is defined for a component, it leaves all that component's views as they were before. That is, all values which were visible remain visible and no new values become visible. When an action is defined for a component, but that action returns undef, no new values become visible, and all descendant values which were visible disappear.

Root Element versus :TOP Pseudoclass

It is important to understand the very special function of the :TOP component, and to avoid confusing it with the HTML root element. The most important distinctions are that

The semantic action for :TOP psuedoclass is always the last action to be called in a parse.
The :TOP component is always the entire HTML document. This can be true of the root element, but it is not true in all cases.
The value that the action for the :TOP component returns becomes the value that the Marpa::UrHTML::urhtml method returns.

The root element is the HTML element whose tagname is "html", though its start and end tags are optional and can be omitted even in strictly valid HTML. Tags or no tags, every HTML document has a root element. (The :TOP component is not an element, so it does not have a tagname and never has tags.)

The root element is always a descendant of the :TOP component. The SGML prolog and SGML trailer are always descendants of the :TOP component. The SGML prolog and SGML trailer are never descendants of the root element.

If an action for the root element is specified, it will also be called once and only once in every parse. An action for the root element can be specified in same way as actions for other elements, by using its tagname of "html". An element wildcard action also becomes the action for the root element, if no more specific action takes precedence.

A :TOP action will be called once and only once in every parse. The :TOP action is unique in that there is a default action -- that is, an action is run whether the user specifies it or not. No other component has a default action.

Tags versus Structure

In a defective HTML document, tags for an element may be missing. Also, tags for an element may exist that did not fit into the structure of an HTML document, even by HTML::Parser's very liberal standards.

Where tags conflict with structure, HTML::Parser follows structure. Following the structure means, for example, that handlers for the html, head, and body elements will be called once and only once for every document.

For example, consider this short and very defective HTML document:

<title>Short</title><p>Text</head><head>

HTML::Parser starts the HTML document's body when it encounters the <p> start tag. That means that, even if they were in the right order, the two head tags cannot be fit into any reasonable parse structure.

If an action is specified for the head element, it will be called for the actual header, and the original view of the head element component will be the text "<title>Short</title>". The action for the head element will not be called again. The two stray tags, </head> and <head>, will be treated as descendants of the body element, and reclassed as "cruft" terminals.

Explicit and Implicit Elements

If a handler is specified for a tagname, it is called whenever that element is found, even if there are no explicit tags for that element. The HTML standards allow both tags for html, head, body and tbody to be missing. Marpa::UrHTML is more liberal, and will recognize virtual tags for table, tr, and td elements as required to repair a defective table.

Marpa::UrHTML is more even liberal about recognizing virtual end tags than it is about start tags. Virtual start tags are recognized only for the specific elements listed above. For any non-empty HTML element, there is some circumstance under which Marpa::UrHTML will recognize a virtual end tag. As one example, at end of file Marpa::UrHTML will do its best to produce a balanced HTML structure by creating a virtual end tag for every element in the stack of currently active elements.

EXPORTS

Marpa::UrHTML exports nothing by default. Optionally, Marpa::UrHTML::urhtml may be exported.

LICENSE AND COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl 5.10.0.

To install Marpa::UrHTML, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Marpa::UrHTML

CPAN shell

perl -MCPAN -e shell
install Marpa::UrHTML

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)