NAME

HTML::TableExtract - Perl module for extracting the text contained in tables within an HTML document.

SYNOPSIS

# Matched tables are returned as "table state" objects; tables can be
# matched using column headers, depth, count within a depth, table tag
# attributes, or some combination of the four.

# Using column header information. Assume an HTML document with tables
# that have "Date", "Price", and "Cost" somewhere in a row. The columns
# beneath those headings are what you want to extract. They will be
# returned in the same order as you specified the headers since
# 'automap' is enabled by default.

use HTML::TableExtract;
$te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] );
$te->parse($html_string);

# Examine all matching tables
foreach $ts ($te->table_states) {
  print "Table (", join(',', $ts->coords), "):\n";
  foreach $row ($ts->rows) {
     print join(',', @$row), "\n";
  }
}

# Shorthand...top level rows() method assumes the first table found in
# the document if no arguments are supplied.
foreach $row ($te->rows) {
   print join(',', @$row), "\n";
}

# Using depth and count information. Every table in the document has
# a unique depth and count tuple, so when both are specified it is a
# unique table. Depth and count both begin with 0, so in this case
# we are looking for a table (depth 2) within a table (depth 1)
# within a table (depth 0, which is the top level HTML document). In
# addition, it must be the third (count 2) such instance of a table
# at that depth.

$te = HTML::TableExtract->new( depth => 2, count => 2 );
$te->parse_file($html_file);
foreach $ts ($te->table_states) {
   print "Table found at ", join(',', $ts->coords), ":\n";
   foreach $row ($ts->rows) {
      print "   ", join(',', @$row), "\n";
   }
}

# Using table tag attributes. If multiple attributes are specified, all
# must be present and equal for match to occur.

$te = HTML::TableExtract->new( attribs => { border => 1 } );
$te->parse($html_string);
foreach $ts ($te->table_states) {
  print "Table with border=1 found at ", join(',', $ts->coords), ":\n";
  foreach $row ($ts->rows) {
     print "   ", join(',', @$row), "\n";
  }
}
   

DESCRIPTION

HTML::TableExtract is a subclass of HTML::Parser that serves to extract the textual information from tables of interest contained within an HTML document. The text from each extracted table is stored in tabe state objects which hold the information as an array of arrays that represent the rows and cells of that table.

There are three constraints available to specify which tables you would like to extract from a document: Headers, Depth, Count, and Attributes.

Headers, the most flexible and adaptive of the techniques, involves specifying text in an array that you expect to appear above the data in the tables of interest. Once all headers have been located in a row of that table, all further cells beneath the columns that matched your headers are extracted. All other columns are ignored: think of it as vertical slices through a table. In addition, TableExtract automatically rearranges each row in the same order as the headers you provided. If you would like to disable this, set automap to 0 during object creation, and instead rely on the column_map() method to find out the order in which the headers were found. Furthermore, TableExtract will automatically compensate for cell span issues so that columns are really the same columns as you would visually see in a browser. This behavior can be disabled by setting the gridmap parameter to 0. HTML is stripped from the entire textual content of a cell before header matches are attempted -- unless the keep_html parameter was enabled.

Depth and Count are more specific ways to specify tables in relation to one another. Depth represents how deeply a table resides in other tables. The depth of a top-level table in the document is 0. A table within a top-level table has a depth of 1, and so on. Each depth can be thought of as a layer; tables sharing the same depth are on the same layer. Within each of these layers, Count represents the order in which a table was seen at that depth, starting with 0. Providing both a depth and a count will uniquely specify a table within a document.

Attributes match based on the attributes of the html <table> tag, for example, boder widths or background color.

Each of the Headers, Depth, Count, and Attributes specifications are cumulative in their effect on the overall extraction. For instance, if you specify only a Depth, then you get all tables at that depth (note that these could very well reside in separate higher- level tables throughout the document since depth extends across tables). If you specify only a Count, then the tables at that Count from all depths are returned (i.e., the nth occurrence of a table at each depth). If you only specify Headers, then you get all tables in the document containing those column headers. If you have specified multiple constraints of Headers, Depth, Count, and Attributes, then each constraint has veto power over whether a particular table is extracted.

If no Headers, Depth, Count, or Attributes are specified, then all tables match.

Text that is gathered from the tables is decoded with HTML::Entities by default; this can be disabled by setting the decode parameter to .

Chains

Make sure you fully understand the notions of depth and count before proceeding, because it is about to become a bit more involved.

Table matches using Headers, Depth, Count, or Attributes can be chained together in order to further specify tables relative to one another. Links in chains are successively applied to tables within tables. Top level constraints (i.e., header, depth, and count parameters for the TableExtract object) behave as the first link in the chain. Additional links are specified using the chain parameter. Each link in the chain has its own set of constraints. For example:

$te = HTML::TableExtract->new(
   headers => [qw(Summary Region)],
   chain   => [
               { depth => 0, count => 2 },
               { headers => [qw(Part Qty Cost)] }
              ],
);

The matching process in this case will start with all tables in the document that have "Summary" and "Region" in their headers. For now, assume that there was only one table that matched these headers. Each table contained within that table will be compared to the first link in the chain. Depth 0 means that a matching table must be immediately contained within the current table; count 2 means that the matching table must also be the third at that depth (counts and depths start at ). In other words, the next link of the chain will match on the third table immediately contained within our first matched table. Once this link matches, then all further tables beneath that table that have "Part", "Qty", and "Cost" in their headers will match. By default, it is only tables at the end of the chains that are returned to the application, so these tables are returned.

Each time a link in a chain matches a table, an additional context for depth and count is established. It is perhaps easiest to visualize a context as a brand-new HTML document, with new depths and counts to compare to the remaining links in the chain. The top level HTML document is the first context. Each table in the document establishes a new context. Depth in a chain link is relative to the context that the matching table creates (i.e., a link depth of 0 would be a table immediately contained within the table that matched the prior link in the chain). Likewise, that same context keeps track of counts within the new depth scheme for comparison to the remaining links in the chain. Headers still apply if they are present in a link, but they are always independent of context.

As it turns out, specifying a depth and count provides a unique address for a table within a context. For non-unique constraints, such as just a depth, or headers, there can be multiple matches for a given link. In these cases the chain "forks" and attempts to make further matches within each of these tables.

By default, chains are elastic. This means that when a particular link does not match on a table, it is passed down to subtables unchanged. For example:

$te = HTML::TableExtract->new(
   headers => [qw(Summary Region)],
   chain   => [
               { headers => [qw(Part Qty Cost)] }
              ],
);

If there are intervening tables between the two header queries, they will be ignored; this query will extract all tables with "Part", "Qty", and "Cost" in the headers that are contained in any table with "Summary" and "Region" in its headers, regardless of how embedded the inner tables are. If you want a chain to be inelastic, you can set the elastic parameter to 0 for the whole TableExtract object. Using the same example:

$te = HTML::TableExtract->new(
   headers => [qw(Summary Region)],
   chain   => [
               { headers => [qw(Part Qty Cost)] }
              ],
   elastic => 0,
);

In this case, the inner table (Part, Qty, Cost) must be immediately contained within the outer table (Summary, Region) in order for the match to take place. This is equivalent to specifying a depth of 0 for each link in the chain; if you only want particular links to be inelastic, then simply set their depths to 0.

By default, only tables that match at the end of the chains are retained. The intermediate matches along the chain are referred to as waypoints, and are not extracted by default. A waypoint may be retained, however, by specifiying the keep parameter in that link of the chain. This parameter may be specified at the top level as well if you want to keep tables that match the first set of constraints in the object. If you want to keep all tables that match along the chain, the specify the keepall parameter at the top level.

Are chains overkill? Probably. In reality, nested HTML tables tend not to be very deep, so there will usually not be much need for lots of links in a chain. Theoretically, however, chains offer precise targeting of tables relative to one another, no matter how deeply nested they are.

Pop Quiz

What happens with the following table extraction?

$te = HTML::TableExtract->new(
  chain => [ { depth => 0 } ],
);

Answer: All tables that are contained in another table are extracted from the document. In this case, there were no top-level constraints specified, which if you recall means that all tables match the first set of constraints (or non-constraints, in this case!). A depth of 0 in the next link of the chain means that the matching table must be immediately contained within the table from a prior match.

The following is equivalent:

$te = HTML::TableExtract->new(
  depth     => 1,
  subtables => 1,
);

The subtables parameter tells TableExtract to scoop up all tables contained within the matching tables. In conjunction with a depth of 1, this has the affect of discarding all top-level tables in the document, which is exactly what occurred in the prior example.

Advice

The main point of this module was to provide a flexible method of extracting tabular information from HTML documents without relying to heavily on the document layout. For that reason, I suggest using Headers whenever possible -- that way, you are anchoring your extraction on what the document is trying to communicate rather than some feature of the HTML comprising the document (other than the fact that the data is contained in a table).

HTML::TableExtract is a subclass of HTML::Parser, and as such inherits all of its basic methods. In particular, start(), end(), and text() are utilized. Feel free to override them, but if you do not eventually invoke them in the SUPER class with some content, results are not guaranteed.

METHODS

The following are the top-level methods of the HTML::TableExtract object. Tables that have matched a query are actually returned as separate objects of type HTML::TableExtract::TableState. These table state objects have their own methods, documented further below. There are some top-level methods that are present for convenience and backwards compatibility that are nothing more than front-ends for equivalent table state methods.

CONSTRUCTOR

new()

Return a new HTML::TableExtract object. Valid attributes are:

headers

Passed as an array reference, headers specify strings of interest at the top of columns within targeted tables. These header strings will eventually be passed through a non-anchored, case-insensitive regular expression, so regexp special characters are allowed. The table row containing the headers is not returned. Columns that are not beneath one of the provided headers will be ignored. Columns will, by default, be rearranged into the same order as the headers you provide (see the automap parameter for more information). Additionally, by default columns are considered what you would see visually beneath that header when the table is rendered in a browser. See the gridmap parameter for more information. HTML within a header is stripped before the match is attempted, unless the keep_html parameter was specified.

depth

Specify how embedded in other tables your tables of interest should be. Top-level tables in the HTML document have a depth of 0, tables within top-level tables have a depth of 1, and so on.

count

Specify which table within each depth you are interested in, beginning with 0.

attribs

Passed as a hash reference, attribs specify attributes of interest within the HTML <table> tag itself.

chain

List of additional constraints to be matched sequentially from the top level constraints. This is a reference to an array of hash references. Each hash is a link in the chain, and can be specified in terms of depth, count, and headers. Further modifiers include keep, which means to retain the table if it would normally be dropped as a waypoint.

automap

Automatically applies the ordering reported by column_map() to the rows returned by rows(). This only makes a difference if you have specified Headers and they turn out to be in a different order in the table than what you specified. Automap will rearrange the columns in the same order as the headers appear. To get the original ordering, you will need to take another slice of each row using column_map(). automap is enabled by default.

slice_columns

Enabled by default, this option controls whether vertical slices are returned from under headers that match. When disabled, all columns of the matching table are retained, regardles of whether they had a matching header above them.

keep_headers

Disabled by default, and only applicable when header constraints have been specified, keep_headers will retain the matching header row as the first row of table data when enabled. The actual header text displayed is subject to the strip_html_on_match parameter below. See also the table state method hrow() below.

gridmap

Controls whether the table contents are returned as a grid or a tree. ROWSPAN and COLSPAN issues are compensated for, and columns really are columns. Empty phantom cells are created where they would have been obscured by ROWSPAN or COLSPAN settings. This really becomes an issue when extracting columns beneath headers. Enabled by default.

keepall

Keep all tables that matched along a chain, including tables matched by top level contraints. By default, waypoints are dropped and only the matches at the end of the chain are retained. To retain a particular waypoint along a chain, use the keep parameter in that link.

elastic

When set to 0, all links in chains will be treated as though they had a depth of 0 specified, which means there can be no intervening unmatched tables between matches on links.

subtables

Extract all tables within matched tables.

decode

Automatically decode retrieved text with HTML::Entities::decode_entities(). Enabled by default.

br_translate

Translate <br> tags into newlines. Sometimes the remaining text can be hard to parse if the <br> tag is simply dropped. Enabled by default. Has no effect if keep_html is enabled.

keep_html

Return the raw HTML contained in the cell, rather than just the visible text. Embedded tables are not retained in the HTML extracted from a cell. Patterns for header matches must take into account HTML in the string if this option is enabled.

strip_html_on_match

When keep_html is enabled, HTML is retained by default during attempts at matching header strings. With strip_html_on_match enabled, html tags are first stripped from header strings before any comparisons are made. (so if strip_html_on_match is not enabled and keep_html is, you would have to include potential HTML tags in the regexp for header matches). Stripped header tags are replaced with an empty string, e.g. 'hot d<em>og</em>' would become 'hot dog' before attempting a match.

error_handle

Filehandle where error messages are printed. STDERR by default.

debug

Prints some debugging information to STDERR, more for higher values. If error_handle was specified, messages are printed there rather than STDERR.

REGULAR METHODS

The following methods are invoked directly from an HTML::TableExtract object.

depths()

Returns all depths that contained matched tables in the document.

counts($depth)

For a particular depth, returns all counts that contained matched tables.

table_state($depth, $count)

For a particular depth and count, return the table state object for the table found, if any.

table_states()

Return table state objects for all tables that matched. Returns an empty list if no tables matched.

first_table_state_found()

Return the table state object for the first table matched in the document. Returns undef if no tables were matched.

tables_report([$show_content, $col_sep])

Return a string summarizing extracted tables, along with their depth and count. Optionally takes a $show_content flag which will dump the extracted contents of each table as well with columns separated by $col_sep. Default $col_sep is ':'.

tables_dump([$show_content, $col_sep])

Same as tables_report() except dump the information to STDOUT.

TABLE STATE METHODS

The following methods are invoked from an HTML::TableExtract::TableState object, such as those returned from the table_states() method.

rows()

Return all rows within a matched table. Each row returned is a reference to an array containing the text of each cell.

depth()

Return the (absolute) depth at which this table was found.

count()

Return the count for this table within the depth it was found.

coords()

Return depth and count in a list.

hrow()

Returns the header row as a list when headers were specified as a constraint. If keep_headers was specified initially, this is equivalent to the first row returned by the rows() method.

column_map()

Return the order (via indices) in which the provided headers were found. These indices can be used as slices on rows to either order the rows in the same order as headers or restore the rows to their natural order, depending on whether the rows have been pre-adjusted using the automap parameter.

lineage()

Returns the path of matched tables that led to matching this table. The path is a list of array refs containing depth, count, row, and column values for each ancestor table involved. Note table state objects will not exist for ancestral tables that did not match any criterion.

DEPRECATED SUBS/METHODS

The following methods are depracated, old-style subroutines. Eventually they will do what their corresponding *state() methods will do. For example, rather than using table_states(), in the future you can just use tables(). Currently tables() returns a raw array of the table contents.

table($depth, $count)

Will soon do the same as table_state()

tables()

Will soon do the same as table_states()

first_table_found()

Will soon do the same as first_table_state_found()

table_coords($table)

Will soon go away completely. Instead use $t->coords()

rows()

rows($table)

Will soon go away completely. Instead use $t->rows()

column_map($table)

Will soon go away completely. Instead use $t->column_map()

REQUIRES

HTML::Parser(3), HTML::Entities(3)

AUTHOR

Matthew P. Sisk, <sisk@mojotoad.com>

COPYRIGHT

Copyright (c) 2000-2005 Matthew P. Sisk. All rights reserved. All wrongs revenged. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

HTML::Parser(3), perl(1).

cpanm HTML::TableExtract

perl -MCPAN -e shell
install HTML::TableExtract