Revision history for Perl module WWW-3172-Crawler

0.001     2011-10-03
 - First stab at the crawler
    - Required stats: meta description and keywords
    - Extra stats: load time, size
    - Extra features: Obeys robots.txt, crawls from most linked-to to least