NAME

WWW::RobotRules::Parser::MultiValue - Parse robots.txt

SYNOPSIS

use WWW::RobotRules::Parser::MultiValue;
use LWP::Simple qw(get);

my $url = 'http://example.com/robots.txt';
my $robots_txt = get $url;

my $rules = WWW::RobotRules::Parser::MultiValue->new(
    agent => 'TestBot/1.0',
);
$rules->parse($url, $robots_txt);

if ($rules->allows('http://example.com/some/path')) {
    my $delay = $rules->delay_for('http://example.com/');
    sleep $delay;
    ...
}

my $hash = $rules->rules_for('http://example.com/');
my @list_of_allowed_paths = $hash->get_all('allow');
my @list_of_custom_rule_value = $hash->get_all('some-rule');

DESCRIPTION

WWW::RobotRules::Parser::MultiValue is a parser for robots.txt.

Parsed rules for the specified user agent is stored as a Hash::MultiValue, where the key is a lower case rule name.

Request-rate rule is handled specially. It is normalized to Crawl-delay rule.

METHODS

• new

    $rules = WWW::RobotRules::Parser::MultiValue->new(
        aget => $user_agent
    );
    $rules = WWW::RobotRules::Parser::MultiValue->new(
        aget => $user_agent,
        ignore_default => 1,
    );
  

  Creates a new object to handle rules in robots.txt. The object parses rules match with $user_agent. The rules of User-agent: * always match and have a lower precedence than the rules explicitly matched with $user_agent. If ignore_default option is specified, rules of User-agent: * are simply ignored.

• parse

    $rules->parse($uri, $text);
  

  Parses a text content $text whose URI is $uri.

• match_ua

    $rules->match_ua($pattern);
  

  Test if the user agent matches with $pattern.

• rules_for

    $hash = $rules->rules_for($uri);
  

  Returns a Hash::MultiValue, which describes the rules of the domain of $uri.

• allows

    $test = $rules->allows($uri);
  

  Tests if the user agent is allowed to visit $uri. If there is 'Allow' rule for the path of $uri, then the $uri is allowed to visit. If there is 'Disallow' rule for the path of $uri, then the $uri is not allowed to visit. Otherwise, the $uri is allowed to visit.

• delay_for

    $delay = $rules->delay_for($uri);
    $delay_in_milliseconds = $rules->delay_for($uri, 1000);
  

  Calculate a crawl delay for the specified $uri. The value is determined by 'Crawl-delay' rule or 'Request-rate' rule. The second argument specifies the base of the return value.

SEE ALSO

Hash::MultiValue

LICENSE

Copyright (C) INA Lintaro

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

AUTHOR

INA Lintaro tarao.gnn@gmail.com