/ 
SWISH-Filter-0.191
River stage two • 4 direct dependents • 19 total dependents
/ SWISH::Filter

NAME

SWISH::Filter - filter documents for indexing with Swish-e

SYNOPSIS

use SWISH::Filter;

# load available filters into memory
my $filter = SWISH::Filter->new;


# convert a document

my $doc = $filter->convert(
      document     => \$scalar_ref,  # path or ref to a doc
      content_type => $content_type, # content type if doc reference
      name         => $real_path,    # optional name for this file (useful for debugging)
      user_data    => $whatever,     # optional data to make available to filters
 );

return unless $doc;  # empty doc, zero size, or no filters installed

# Was the document converted by a filter?
my $was_filtered = $doc->was_filtered;

# Skip if the file is not text
return if $doc->is_binary;

# Print out the doc
my $doc_ref = $doc->fetch_doc;
print $$doc_ref;

# Fetch the final content type of the document
my $content_type = $doc->content_type;

# Fetch Swish-e parser type (TXT*, XML*, HTML*, or undefined)
my $doc_type = $doc->swish_parser_type;

DESCRIPTION

SWISH::Filter provides a unified way to convert documents into a type that Swish-e can index. Individual filters are installed as separate subclasses (modules). For example, there might be a filter that converts from PDF format to HTML format.

SWISH::Filter is a framework that relies on other packages to do the heavy lifting of converting non-text documents to text. Additional helper programs or Perl modules may need to be installed to use SWISH::Filter to filter documents. For example, to filter PDF documents you must install the Xpdf package.

The filters are automatically loaded when SWISH::Filters->new() is called. Filters define a type and priority that determines the processing order of the filter. Filters are processed in this sort order until a filter accepts the document for filtering. The filter uses the document's content type to determine if the filter should handle the current document. The content-type is determined by the files suffix if not supplied by the calling program.

The individual filters are not designed to be used as separate modules. All access to the filters is through this SWISH::Filter module.

Normally, once a document is filtered processing stops. Filters can filter the document and then set a flag saying that filtering should continue (for example a filter that uncompresses a MS Word document before passing on to the filter that converts from MS Word to text). All this should be transparent to the end user. So, filters can be pipe-lined.

The idea of SWISH::Filter is that new filters can be created, and then downloaded and installed to provide new filtering capabilities. For example, if you needed to index MS Excel documents you might be able to download a filter from the Swish-e site and magically next time you run indexing MS Excel docs would be indexed.

The SWISH::Filter setup can be used with -S prog or -S http. It works best with the -S prog method because the filter modules only need to be loaded and compiled one time. The -S prog program spider.pl will automatically use SWISH::Filter when spidering with default settings (using "default" as the first parameter to spider.pl).

The -S http indexing method uses a Perl helper script called swishspider. swishspider has been updated to work with SWISH::Filter, but (unlike spider.pl) does not contain a "use lib" line to point to the location of SWISH::Filter. This means that by default swishspider will not use SWISH::Filter for filtering. The reason for this is because swishspider runs for every URL fetched, and loading the Filters for each document can be slow. The recommended way of spidering is using -S prog with spider.pl, but if -S http is desired the way to enable SWISH::Filter is to set PERL5LIB before running swish so that swishspider will be able to locate the SWISH::Filter module. Here's one way to set the PERL5LIB with the bash shell:

$ export PERL5LIB=`swish-filter-test -path`

METHODS

new( %opts )

new() creates a SWISH::Filter object. You may pass in options as a list or a hash reference.

Options

There is currently only one option that can be passed in to new():

ignore_filters

Pass in a reference to a list of filter names to ignore. For example, if you have two filters installed "Pdf2HTML" and "Pdf2XML" and want to avoid using "Pdf2XML":

my $filter = SWISH::Filter->new( ignore_filters => ['Pdf2XML'];

doc_class

If you subclass SWISH::Filter::Document with your own class, indicate your class name in the new() method with the doc_class param. The return value of doc_class() is used in convert() for instatiating the Document object. The default value is SWISH::Filter::Document.

convert

This method filters a document. Returns an object belonging to doc_class() on success. If passed an empty document, a filename that cannot be read off disk, or if no filters have been loaded, returns undef.

See the SWISH::Filter::Document documentation.

You must pass in a hash (or hash reference) of parameters to the convert() method. The possible parameters are:

document

This can be either a path to a file, or a scalar reference to a document in memory. This is required.

content_type

The MIME type of the document. This is only required when passing in a scalar reference to a document. The content type string is what the filters use to match a document type.

When passing in a file name and content_type is not set, then the content type will be determined from the file's extension by using the MIME::Types Perl module (available on CPAN).

name

Optional name to pass in to filters that will be used in error and warning messages.

user_data

Optional data structure that all filters may access. This can be fetched in a filter by:

my $user_data = $doc_object->user_data;

And used in the filter as:

if ( ref $user_data && $user_data->{pdf2html}{title} ) {
   ...
}

It's up to the filter author to use a unique first-level hash key for a given filter.

meta_data

Optional data structure intended for meta name/content pairs for HTML or XML output. See SWISH::Filter::Document for discussion of this data.

Example of using the convert() method:

$doc_object = $filter->convert(
    document     => $doc_ref,
    content-type => 'application/pdf',
);

mywarn

Internal method used for writing warning messages to STDERR if $ENV{FILTER_DEBUG} is set. Set the environment variable FILTER_DEBUG before running to see extra messages while processing.

filter_list

Returns a list of filter objects installed.

can_filter( content_type )

This is useful for testing to see if a mimetype might be handled by SWISH::Filter wihtout having to pass in a document. Helpful if doing HEAD requests.

Returns an array of filters that can handle this type of document

decode_content_type( filename )

Returns MIME type for filename if known.

WRITING FILTERS

Filters are standard perl modules that are installed into the SWISH::Filters name space. Filters are not complicated -- see the core SWISH::Filters::* modules for examples.

Each filter defines the content-types (or mimetypes) that it can handle. These are specified as a list of regular expressions to match against the document's content-type. If one of the mimetypes of a filter match the incoming document's content-type the filter is called. The filter can then either filter the content or return undefined indicating that it decided not to filter the document for some reason. If the document is converted the filter returns either a reference to a scalar of the content or a file name where the content is stored. The filter also must change the content-type of the document to reflect the new document.

Filters typically use external programs or modules to do that actual work of converting a document from one type to another. For example, programs in the Xpdf packages are used for converting PDF files. The filter can (and should) test for those programs in its new() method.

Filters also can define a type and priority. These attributes are used to set the order filters are tested for a content-type match. This allows you to have more than one filter that can work on the same content-type. A lower priority value is given preference over a higher priority value.

If a filter calls die() then the filter is removed from the chain and will not be called again during the same run. Calling die when running with -S http or -S fs has no effect since the program is run once per document.

Once a filter returns something other than undef no more filters will be called. If the filter calls $filter->set_continue then processing will continue as if the file was not filtered. For example, a filter can uncompress data and then set $filter->set_continue and let other filters process the document.

A filter may define the following methods (required methods are indicated):

new() required

This method returns either an object which provides access to the filter, or undefined if the filter is not to be used.

The new() method is a good place to check for required modules or helper programs. Returning undefined prevents the filter from being included in the filter chain.

The new method must return a blessed hash reference. The only required attribute is mimetypes. This attribute must contain a reference to an array of regular expressions used for matching the content-type of the document passed in.

Example:

sub new {
    my ( $class ) = @_;

    # List of regular expressions
    my @mimetypes = (
        qr[application/(x-)?msword],
        qr[application/worddoc],
    );

    my %settings = (
        mimetypes   => \@mimetypes,

        # Optional settings
        priority    => 20,
        type        => 2,
    );

    return bless \%settings, $class;
}

The attribute "mimetypes" returns an array reference to a list of regular expressions. Those patterns are matched against each document's content type.

filter() required

This is the function that does the work of converting a document from one content type to another. The function is passed the document object. See document object methods listed below for what methods may be called on a document.

The function can return undefined (or any false value) to indicate that the filter did not want to process the document. Other filters will then be tested for a content type match.

If the document is filtered then the filter must set the new document's content type (if it changed) and return either a file name where the document can be found or a reference to a scalar containing the document.

The filter() method may also return a second value for storing metadata. The value is typically a hash ref of name/value pairs. This value can then be accessed via the meta_data() method in the SWISH::Filter::Document class.

type()

Returns a number. Filters are sorted (for processing in a specific order) and this number is simply the primary key used in sorting. If not specified the filter's type used for sorting is 2.

This is an optional method. You can also set the type in your new() constructor as shown above.

priority()

Returns a number. Filters are sorted (for processing in a specific order) and this number is simply the secondary key used in sorting. If not specified the filter's priority is 50.

This is an optional method. You can also set the priority in your new() constructor as shown above.

Again, the point of the type() and priority() methods is to allow setting the sort order of the filters. Useful if you have two filters for filtering the same content-type, but prefer to use one over the other. Neither are required.

EXAMPLE FILTER

Here's a module to convert MS Word documents using the program "catdoc":

package SWISH::Filters::Doc2txt;
use vars qw/ $VERSION /;

$VERSION = '0.191';


sub new {
    my ( $class ) = @_;

    my $self = bless {
        mimetypes   => [ qr!application/(x-)?msword! ],
        priority    => 50,
    }, $class;


    # check for helpers
    return $self->set_programs( 'catdoc' );

}


sub filter {
    my ( $self, $doc ) = @_;

    my $content = $self->run_catdoc( $doc->fetch_filename ) || return;

    # update the document's content type
    $filter->set_content_type( 'text/plain' );

    # return the document
    return \$content;
}
1;

The new() constructor creates a blessed hash which contains an array reference of mimetypes patterns that this filter accepts. The priority sets this filter to run after any other filters that might handle the same type of content. The set_programs() function says that we need to call a program called "catdoc". The function either returns $self or undefined if catdoc could not be found. The set_programs() function creates a new method for running catdoc.

The filter function runs catdoc passing in the name of the file (if the file is in memory a temporary file is created). That run_catdoc() function was created by the set_programs() call above.

TESTING

Filters can be tested with the swish-filter-test program in the example/ directory. Run:

swish-filter-test -man

for documentation.

TODO

The File::Extract package on CPAN does much of the same work as SWISH::Filter, but used more native Perl. It might be worth investigating if there is anything to be gained by using it in any of the core filters.

SUPPORT

Please contact the Swish-e discussion list. http://swish-e.org

AUTHOR

Bill Moseley

Currently maintained by Peter Karman perl@peknet.com

COPYRIGHT

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.