Changes for version 0.08004
- General
- Previously block_private_address was blocking all addreses. Fixed (Kazuho Oku)
- Previously private addresses were only checked after a DNS resolution. Now given an address that contains IP addresses to begin with are also checked (Kazuho Oku)
- Local addresses other than loopback (127.0.0.1) are also checked.
- POE Engine
- Parameters can now be passed to PoCo::Client::DNS (Kazuho Oku)
- Build
- Retooled the tests.
- Fixed requires list.
Documentation
An Extensible, High-Performance Web Crawler Framework
Modules
Yet Another High Performance Web Crawler Framework
Base Class For Various Gungho Objects
Component Base Class For Gungho
Base Class For WWW Authentication
Add Basic Auth To Gungho
Respect robots.txt
A Rule Object
RobotRules Storage Base Class
DB_File Storage For RobotRules
Base Class To Throttle Requests
Throttle By Domain
Throttle By Number Of Requests
Data::Throttler Based Throttling
Base Class For Gungho Engine
Gungho Engine Using Danga::Socket
IO::Async Engine
POE Engine For Gungho
Gungho Exceptions
Base Class For Gungho Handlers
Write Out Fetched Contents To File
A Handler That Does Nothing
Inline Your Providers And Handlers
Gungho Log Class
Gungho Plugin Base Class
Keep Track Of Time To Finish Request
Base Class For Gungho Prividers
Provide Requests From A Simple File
An In-Memory, Simple Provider
Specify requests in YAML format
A Gungho Request Object
HTTP specific utilities
Provides
in lib/Gungho/Engine/IO/Async.pm
in lib/Gungho/Inline.pm
in lib/Gungho/Inline.pm
Examples
- examples/robotrules/simple.yml
- examples/simple-file/simple-file.yml
- examples/simple-file/url.txt
- examples/simple-write-to-file/simple-write-to-file.yml
- examples/simple/simple.yml
- examples/throttle-simple/throttle-simple.yml
- examples/throttle-simple/url.txt
- examples/yaml/config.yml
- examples/yaml/url.yml