/ 
HTML-TokeParser-Simple-1.1
River stage two • 30 direct dependents • 93 total dependents
/ HTML::TokeParser::Simple

NAME

HTML::TokeParser::Simple - easy to use HTML::TokeParser interface

SYNOPSIS

use HTML::TokeParser::Simple;
my $p = HTML::TokeParser::Simple->new( $somefile );

while ( my $token = $parser->get_token )
{
    # This prints all text in an HTML doc (i.e., it strips the HTML)
    next if ! $parser->is_text( $token );
    print $parser->return_text( $token );
}

DESCRIPTION

HTML::TokeParser is a fairly common method of parsing HTML. However, the tokens returned are not exactly intuitive to parse:

["S",  $tag, $attr, $attrseq, $text]
["E",  $tag, $text]
["T",  $text, $is_data]
["C",  $text]
["D",  $text]
["PI", $token0, $text]

To simplify this, HTML::TokeParser::Simple allows the user ask more intuitive (read: more self-documenting) questions about the tokens returned. Specifically, there are 6 is_foo type methods and 5 return_bar type methods. The is_ methods allow you to determine the token type and the return_ methods get the data that you need.

Since this is a subclass of HTML::TokeParser, all HTML::TokeParser methods are available. To truly appreciate the power of this module, please read the documentation for HTML::TokeParser and HTML::Parser.

The following will be brief descriptions of the available methods followed by examples.

is_ Methods

Note:

1 is_start_tag

Use this to determine if you have a start tag. An optional "tag type" may be passed. This will allow you to match if it's a particular start tag. The supplied tag is case-insensitive.

if ( $token->is_start_tag( 'font' ) ) { ... }
2 is_end_tag.

Use this to determine if you have an end tag. An optional "tag type" may be passed. This will allow you to match if it's a particular end tag. The supplied tag is case-insensitive.

Note: due to the way that HTML::TokeParser handles tags, the is_end_tag method for tokens returned with the get_tag() method will have end tags with a forward slash on the front (e.g. /a), whereas tokens returned by the get_token() method will not have this slash. I considered making them the same, but decided to stick with the original format so as to avoid confusion. This is a bit confusing, though, and it may change in future versions.

while ( $token = $p->get_token ) {
  if ( $token->is_end_tag( 'form' ) ) { ... }
}

# or

while ( $token = $p->get_tag ) {
  if ( $token->is_end_tag( '/form' ) ) { ... }
}
3 is_text

Use this to determine if you have text. Note that this is not to be confused with the return_text method described below! is_text will identify text that the user typically sees display in the Web browser.

4 is_comment

Are you still reading this? Nobody reads POD. Don't you know you're supposed to go to CLPM, ask a question that's answered in the POD and get flamed? It's a rite of passage.

Really.

is_comment is used to identify comments. See the HTML::Parser documentation for more information about comments. There's more than you might think.

5 is_declaration

This will match the DTD at the top of your HTML. (You do use DTD's, don't you?)

6 is_process_instruction

Process Instructions are from XML. This is very handy if you need to parse out PHP and similar things with a parser.

The return_ methods

Note:

In case it's not blindingly obvious (I've been bitten by this myself when writing the tests), you should generally test what type of token you have before you call some return_ methods. For example, if you have an end tag, there is no point in calling the return_attrseq method. Calling an innapropriate method will return an empty string.

As noted for the is_ methods, these methods are case-insensitive after the return_ part.

1 return_tag

Do you have a start tag or end tag? This will return the type (lower case).

2 return_attr

If you have a start tag, this will return a hash ref with the attribute names as keys and the values as the values.

3 return_attrseq

For a start tag, this is an array reference with the sequence of the attributes, if any.

4 return_text

This is the exact text of whatever the token is representing.

5 return_token0

For processing instructions, this will return the token found immediately after the opening tag. Example: For <?php, "php" will be the start of the returned string.

Examples

Finding comments

For some strange reason, your Pointy-Haired Boss (PHB) is convinced that the graphics department is making fun of him by embedding rude things about him in HTML comments. You need to get all HTML comments from the HTML.

use strict;
use HTML::TokeParser::Simple;

my @html_docs = glob( "*.html" );

open PHB, "> phbreport.txt" or die "Cannot open phbreport for writing: $!";

foreach my $doc ( @html_docs )
{
    print "Processing $doc\n";
    my $p = HTML::TokeParser::Simple->new( $doc );
    while ( my $token = $p->get_token )
    {
        next if ! $token->is_comment;
        print PHB $token->return_text, "\n";
    }
}

close PHB;

Stripping Comments

Uh oh. Turns out that your PHB was right for a change. Many of the comments in the HTML weren't very polite. Since your entire graphics department was just fired, it falls on you need to strip those comments from the HTML.

use strict;
use HTML::TokeParser::Simple;

my $new_folder = 'no_comment/';
my @html_docs  = glob( "*.html" );

foreach my $doc ( @html_docs )
{
    print "Processing $doc\n";
    my $new_file = "$new_folder$doc";

    open PHB, "> $new_file" or die "Cannot open $new_file for writing: $!";

    my $p = HTML::TokeParser::Simple->new( $doc );
    while ( my $token = $p->get_token )
    {
        next if $token->is_comment;
        print PHB $token->return_text;
    }
    close PHB;
}

Changing form tags

Your company was foo.com and now is bar.com. Unfortunately, whoever wrote your HTML decided to hardcode "http://www.foo.com/" into the action attribute of the form tags. You need to change it to "http://www.bar.com/".

use strict;
use HTML::TokeParser::Simple;

my $new_folder = 'new_html/';
my @html_docs  = glob( "*.html" );

foreach my $doc ( @html_docs )
{
    print "Processing $doc\n";
    my $new_file = "$new_folder$doc";

    open FILE, "> $new_file" or die "Cannot open $new_file for writing: $!";

    my $p = HTML::TokeParser::Simple->new( $doc );
    while ( my $token = $p->get_token )
    {
        if ( $token->is_start_tag( 'form' ) )
        {
            my $form_tag = new_form_tag( $token->return_attr, $token->return_attrseq );
            print FILE $form_tag;
        }
        else
        {
            print FILE $token->return_text;
        }
    }
    close FILE;
}

sub new_form_tag {
    my ( $attr, $attrseq ) = @_;
    if ( exists $attr->{ action } )
    {
        $attr->{ action } =~ s/www\.foo\.com/www.bar.com/;
    }
    my $tag = '';
    foreach ( @$attrseq )
    {
        $tag .= "$_=\"$attr->{ $_ }\" ";
    }
    $tag = "<form $tag>";
    return $tag;
}

HTML::TokeParser::Simple::get_tag()

The HTML::TokeParser::Simple::get_tag() is identical to the get_tag() method in HTML::TokeParser. It returns the next start or end tag in the document, or undef if no more tags are to be had. Thus, you may call any methods on the returned token that you would call on start and end tag tokens returned by the get_token() method. For example, here is the main while loop from a code generator that I use that parses HTML documents and writes a shell of the form parsing code:

while (my $token = $p->get_tag) {
    my $tag = $token->return_tag;

    if ( my $form_pos = ( $tag eq 'form' .. $tag eq '/form' ) ) {
        # Oh!  We're in a form.  Start looking for stuff.
        if ( not_first( $form_pos ) and not_last( $form_pos ) ) {

            add_input_element( $token )  if $tag eq 'input';

            # <select> is a pain, so we need to handle it differently
            if ( my $select_pos = ( $tag eq 'select' .. $tag eq '/select' ) ) {

            # cache select token so we can get its attributes later
            $select_token = $token if $tag eq 'select';

            if ( not_first( $select_pos ) and not_last( $select_pos ) ) {
                    add_select_element( $token, $p, $select_token ) if $tag eq 'option';
                } elsif ( is_last( $select_pos ) ) {
                    # we've finished the <select>, so add it to
                    # %select so we knows we've seen it.
                    $select{ $select_token->return_attr()->{ 'name' } } = '';
                }
            } # end if (select)

            foreach ( qw/ textarea button / ) {
                add_generic_element( $tag, $token ) if $tag eq $_;
            }
        } elsif ( is_last( $form_pos ) ) {
            # we've finished the form, so let's write the document, clear the vars,
            # and start looking for more forms.
            &write_template;
            %element       = ();
            @element_order = ();
            $formnum++;
        }
    } # end if (form)
}

COPYRIGHT

Copyright (c) 2001 Curtis "Ovid" Poe. All rights reserved. This program is free software; you may redistribute it and/or modify it under the same terms as Perl itself

AUTHOR

Curtis "Ovid" Poe poec@yahoo.com

BUGS

2001/10/04 There are no known bugs at this time.

Use of $HTML::Parser::VERSION which is less than 3.25 may result in incorrect behavior as older versions do not always handle XHTML correctly. It is the programmer's responsibility to verify that the behavior of this code matches the programmer's needs.

Address bug reports and comments to: poec@yahoo.com. When sending bug reports, please provide the version of HTML::Parser, HTML::TokeParser, HTML::TokeParser::Simple, the version of Perl, and the version of the operating system you are using.

BUGS

2001/10/04 There are no known bugs at this time.

Use of $HTML::Parser::VERSION which is less than 3.25 may result in incorrect behavior as older versions do not always handle XHTML correctly. It is the programmer's responsibility to verify that the behavior of this code matches the programmer's needs.