There is an ongoing outage on the primary CPAN mirror. It is possible to work around the issue by using MetaCPAN as a mirror.

 / 
CrawlerCommons-RobotRulesParser-0.03
River stage zero No dependents

Changes for version 0.03

  • A version to hopefully get propagated to metacpan as well

Modules

CrawlerCommons::RobotRules
the result of a parsed robots.txt
CrawlerCommons::RobotRulesParser
parser for robots.txt files

Provides

CrawlerCommons::ParseState
in lib/CrawlerCommons/ParseState.pm
CrawlerCommons::RobotDirective
in lib/CrawlerCommons/RobotDirective.pm
CrawlerCommons::RobotRule
in lib/CrawlerCommons/RobotRules.pm
CrawlerCommons::RobotToken
in lib/CrawlerCommons/RobotToken.pm

Other files