Changes for version 0.09000_01
- General
- Backwards Incompatible Change *
- Gungho inheritance has been reimplemented. Components are now called before Gungho::Component::Core, so things work a bit more like normal inheritance.
- Gungho now uses Class::C3::Componentised and does Gungho->load_components()
- Previously, Gungho->setup was the only thing required to setup the framework, but from now both Gungho->bootstrap and Gungho->setup needs to be called
- Private IP blocking has been factored out to Component::BlockPrivateIP. You no longer specify block_private_ip_address setting, but instead you load BlockPrivateIP in the component list. See the documentation for BlockPrivateIP POD
- Refactor Gungho::Base and separate out Gungho:Base::Class
- Engine
- Change Gungho::Engine::POE's default loop_delay to 1
- Scraper
- Add new Scraper component, which allows you to use Web::Scraper from within Gungho
- Log
- Fix calling syntax for Gungho::Log::*::setup
- Tests
- A few bogus tests have been fixed
- POD coverage now ignores RequestTimer
Documentation
An Extensible, High-Performance Web Crawler Framework
Modules
Yet Another High Performance Web Crawler Framework
Base Class For Various Gungho Objects
Base For Classes That Won't Be Instantiated
Component Base Class For Gungho
Base Class For WWW Authentication
Add Basic Auth To Gungho
Block Requests With Private IP Address
Use Cache In Your App
Gungho Core Methods
Respect robots.txt
A Rule Object
RobotRules Storage Base Class
Cache Storage For RobotRules
DB_File Storage For RobotRules
Automatically Parse Robots META
Web::Scraper From Within Gungho
Routines To Setup Gungho
Base Class To Throttle Requests
Throttle By Domain
Throttle By Number Of Requests
Data::Throttler Based Throttling
Base Class For Gungho Engine
Gungho Engine Using Danga::Socket
IO::Async Engine
POE Engine For Gungho
Gungho Exceptions
Base Class For Gungho Handlers
Write Out Fetched Contents To File
A Handler That Does Nothing
Inline Your Providers And Handlers
Log Base Class For Gungho
Log::Dispatch-Based Log For Gungho
Simple Gungho Log Class
Gungho Plugin Base Class
Log Requests
Keep Track Of Time To Finish Request
Base Class For Gungho Prividers
Provide Requests From A Simple File
An In-Memory, Simple Provider
Specify requests in YAML format
A Gungho Request Object
HTTP specific utilities
Gungho HTTP Response Object
Provides
in lib/Gungho/Engine/IO/Async.pm
in lib/Gungho/Inline.pm
in lib/Gungho/Inline.pm
Examples
- examples/robotrules/simple.yml
- examples/simple-file/simple-file.yml
- examples/simple-file/url.txt
- examples/simple-log-dispatch/simple.yml
- examples/simple-write-to-file/simple-write-to-file.yml
- examples/simple/simple.yml
- examples/throttle-simple/throttle-simple.yml
- examples/throttle-simple/url.txt
- examples/yaml/config.yml
- examples/yaml/url.yml