/ 
WWW-CheckSite-0.020
/ WWW::CheckSite::Spider

NAME

WWW::CheckSite::Spider - A base class for spidering the web

SYNOPSIS

use WWW::CheckSite::Spider;

my $sp = WWW::CheckSite::Spider->new(
     uri      => 'http://www.test-smoke.org',
);

while ( my $page = $sp->get_page ) {
    # $page is a hashref with basic information
}

or to spider a site behind HTTP basic authentication:

package BA_Mech;
use base 'WWW::Mechanize';

sub get_basic_credentials { ( 'abeltje', '********' ) }

package main;
use WWW::CheckSite::Spider;

my $sp = WWW::CheckSite::Spider->new(
     ua_class => 'BA_Mech',
     uri      => 'http://your.site.with.ba/',
);

while ( my $page = $sp->get_page ) {
    # $page is a hashref with basic information
}

DESCRIPTION

This module implements a basic web-spider, based on WWW::Mechanize. It takes care of putting pages on the "still-to-fetch" stack. Only uri's with the same origin will be stacked, taking the robots-rules on the server into account.

CONSTATNTS & EXPORTS

The following constants ar exported on demand with the :const tag.

WCS_UNKNOWN
WCS_FOLLOWED
WCS_SPIDERED
WCS_TOSPIDER
WCS_TOFOLLOW
WCS_NOCONTENT
WCS_OUTSCOPE

METHODS

WWW::CheckSite::Spider->new( %opts )

Currently supported options (the rest will be set but not used!):

  • uri => <start_uri> || <\@start_uri> [mandatory]

  • ua_class => by default WWW::Mechanize

  • exclude => <exclude_re> (qr/[#?].*$/)

  • myrules => <\@disallow>

  • lang => languages to pass to Accept-Language: header

$spider->get_page

Fetch the page and do some book keeping. It returns the result of $pider->process_page().

$spider->process_page( $uri )

Override this method to make the spider do something useful. By default it returns:

  • org_uri Used for the request

  • ret_uri The uri returned by the server

  • depth The depth in the browse tree

  • status The return status from server

  • success shortcut for status == 200

  • is_html shortcut for ct eq 'text/html'

  • title What's in the <TITLE></TITLE> section

  • ct The content-type

Filter out the uri's that will fail:

qr!^(?:mailto:|mms://|javascript:)!i

$spider->filter_link( $uri )

Return the URI to be spidered or undef for skipping.

$spider->strip_uri( $uri )

Strip the fragment bit of the $uri.

USERAGENT METHODS

$spider->agent

Retruns a standard name for this UserAgent.

$spider->init_agent

Initialise the agent that is used to fetch pages. The default class is WWW::Mechanize but any class that has the same methods will do.

The ua_class needs to support the following methods (see WWW::Mechanize for more information about these):

new
get
base
uri
status
success
ct
is_html
title
HEAD (for WWW::CheckSite::Validator)
content (for WWW::CheckSite::Validator)
images (for WWW::CheckSite::Validator)

$spider->current_agent

Return the current user agent.

$spider->new_agent

Create a new agent and return it.

ROBOTRULES METHODS

The Spider uses the robot rules mechanism. This means that it will always get the /robots.txt file from the root of the webserver to see if we are allowed (actually "not disallowed") to access pages as a robot.

You can add rules for disallowing pages by specifying a list of lines in the robots.txt syntax to @{ $self->{myrules} }.

$spider->more_rrules( $url )

Check to see if the robots.txt file for this $url has already been loaded. If not, fetch the file and add the rules to the $self->{_r_rules} object.

$spider->uri_ok( $uri )

This will determine whether this uri should be spidered. Rules are simple:

  • Has the same base uri as the one we started with

  • Is not excluded by the $self->{exclude} regex.

  • Is not excluded by robots.txt mechanism

$spider->allowed( $uri )

Checks the uri against the robotrules.

$spider->init_robotrules( )

This will setup a <WWW::RobotRules> object. @{$self->{myrules } is used to add rules and should be in the RobotRules format. These rules are added to the ones found in robots.txt.

$spider->current_rrules

Returns the current RobotRules object.

AUTHOR

Abe Timmerman, <abeltje@cpan.org>

BUGS

Please report any bugs or feature requests to bug-www-checksite@rt.cpan.org, or through the web interface at http://rt.cpan.org. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

COPYRIGHT & LICENSE

Copyright MMV Abe Timmerman, All Rights Reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.