Changes for version 0.001 - 2011-10-03

  • First stab at an interface to the crawler
    • Required stats: meta description and keywords
    • Extra stats: load time, size
    • Extra features: Obeys robots.txt, crawls from most linked-to to least
    • It's pretty!

Documentation

run your WWW::3172::Crawler::WebInterface application

Modules

Provides a web frontend to WWW::3172::Crawler