/ 
HTML-Index-0.05
River stage zero No dependents
/ HTML::Index::Search

NAME

HTML::Index - Perl extension for indexing HTML files

SYNOPSIS

use HTML::Index;

$indexer = HTML::Indexer->new( %options );

$indexer->create_index;

@results = $indexer->search( 
  words => [ 'search', keywords' ],
);

for my $result ( @results )
{
  print "words found: ", $result->words, "\n";
  print "path found on: ", $result->path, "\n";
}

DESCRIPTION

HTML::Index is a simple module for indexing HTML files so that they can be subsequently searched by keywords. It is looselly based on the indexer.pl script in the O'Reilly "CGI Programming with Perl, 2nd Edition" book (http://www.oreilly.com/catalog/cgi2/author.html).

Indexing is based on a list of directories passed to the constructor as one of its options (HTML_DIRS). All files in these directories whose extensions match the EXTENSIONS_REGEX are parsed using HTML::TreeBuilder and the word in those pages added to the index. Words are stored lowercase, anything at least 2 characters long, and consist of alphanumerics ([a-z\d]{2,}).

Indexing is also possible in "remote" mode; here a list of URLs is provided, and indexed files are grabbed via HTTP from these URLs, and all pages linked from them. Only pages on the same site are indexed.

Indexes are stored in various database files. The default is to use Berkeley DB, but the filesystem can be use if Berkeley DB is not installed using Tie::TextDir.

The modification times of files in the index are stored, and they are "re-inexed" if their modification time changes. Searches return results in no particular order - it is up to the caller to re-order them appropriately! Indexes can be run incrementally - only new or updated files will be indexed or re-indexed.

OPTIONS

VERBOSE

Print various bumpf to STDERR.

STOP_WORD_FILE

Specify a file containing "stop words" to ignore when indexling. A sample stopwords.txt file is included in this distribution. MAke sure you use the same STOP_WORD_FILE for indexing and searching. Otherwise, if you submit a search for a word that was in the stop word list when indexing (especially in a combination search) you may not get the result you expect!

REFRESH

Boolean to regenerate the index from scratch.

DB_DIR

Specify a directory to store the Berkeley DB files. Defaults to '.'.

METHODS

create_index

Does exactly what it says on the can.

Search the index, returning an array of HTML::Index::SearchResults objects. Takes two arguments:

AUTHOR

Ave Wrigley <Ave.Wrigley@itn.co.uk>

COPYRIGHT

Copyright (c) 2001 Ave Wrigley. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.