crawlercommons-robotrulesparser

Perl implementation of Google crawler-commons RobotsRulesParser