/ 
Xango-1.08_01

Changes for version 1.08_01

  • Add clear() call to the DNS cache on startup. On systems with prolonged process life time, it's likely that the DNS cache is holding bad data, so we might as well reinitialize.
  • add tools/checkreqs.pl and custom Makefile.PL
  • Fix Push.pm so that shutdown_broker state kills alarms.
  • EXPERIMETAL
    • Try to work with servers that send 400/408 upon too many connections (the sysadmins must hate me) by gracefully reducing the number of HTTP fetchers upon encountering those return status

Documentation

Xango::Manual::Intro
Learn How To Write Crawlers With Xango

Modules

Xango
Event Based High Performance Web Crawler Framework
Xango::Broker
Broker HTTP Requests (Deprecated)
Xango::Broker::Base
Base Class for Xango Broker
Xango::Broker::Pull
Xango's Pull-Crawler
Xango::Broker::Push
Xango's Push-Crawler
Xango::Config
Global Xango Config
Xango::Job
Xango Job

Other files