NAME
AnyEvent::Net::Curl::Queued - Any::Moose wrapper for queued downloads via Net::Curl & AnyEvent
VERSION
version 0.016
SYNOPSIS
#!/usr/bin/env perl
package CrawlApache;
use common::sense;
use HTML::LinkExtor;
use Any::Moose;
extends 'AnyEvent::Net::Curl::Queued::Easy';
after finish => sub {
my ($self, $result) = @_;
say $result . "\t" . $self->final_url;
if (
not $self->has_error
and $self->getinfo('content_type') =~ m{^text/html}
) {
my @links;
HTML::LinkExtor->new(sub {
my ($tag, %links) = @_;
push @links,
grep { $_->scheme eq 'http' and $_->host eq 'localhost' }
values %links;
}, $self->final_url)->parse(${$self->data});
for my $link (@links) {
$self->queue->prepend(sub {
CrawlApache->new({ initial_url => $link });
});
}
}
};
no Any::Moose;
__PACKAGE__->meta->make_immutable;
1;
package main;
use common::sense;
use AnyEvent::Net::Curl::Queued;
my $q = AnyEvent::Net::Curl::Queued->new;
$q->append(sub {
CrawlApache->new({ initial_url => 'http://localhost/manual/' })
});
$q->wait;DESCRIPTION
Efficient and flexible batch downloader with a straight-forward interface:
create a queue;
append/prepend URLs;
wait for downloads to end (retry on errors).
Download init/finish/error handling is defined through Moose's method modifiers.
MOTIVATION
I am very unhappy with the performance of LWP. It's almost perfect for properly handling HTTP headers, cookies & stuff, but it comes at the cost of speed. While this doesn't matter when you make single downloads, batch downloading becomes a real pain.
When I download large batch of documents, I don't care about cookies or headers, only content and proper redirection matters. And, as it is clearly an I/O bottleneck operation, I want to make as many parallel requests as possible.
So, this is what CPAN offers to fulfill my needs:
Net::Curl: Perl interface to the all-mighty libcurl, is well-documented (opposite to WWW::Curl);
AnyEvent: the DBI of event loops. Net::Curl also provides a nice and well-documented example of AnyEvent usage (03-multi-event.pl);
MooseX::NonMoose: Net::Curl uses a Pure-Perl object implementation, which is lightweight, but a bit messy for my Moose-based projects. MooseX::NonMoose patches this gap.
AnyEvent::Net::Curl::Queued is a glue module to wrap it all together. It offers no callbacks and (almost) no default handlers. It's up to you to extend the base class AnyEvent::Net::Curl::Queued::Easy so it will actually download something and store it somewhere.
BENCHMARK
Obviously, the bottleneck of any kind of download agent is the connection itself. However, socket handling and header parsing add a lots of overhead.
The script eg/benchmark.pl compares AnyEvent::Net::Curl::Queued against several other download agents. Only AnyEvent::Net::Curl::Queued itself, AnyEvent::Curl::Multi, Parallel::Downloader and lftp support parallel connections natively; thus, Parallel::ForkManager is used to reproduce the same behaviour for the remaining agents. Both AnyEvent::Curl::Multi and LWP::Curl are frontends for WWW::Curl. Parallel::Downloader uses AnyEvent::HTTP as it's backend.
The download target is a copy of the Apache documentation on an Apache server located in another datacenter. The benchmark separates the client initialization from the download management, when possible (not possible on external utilities neither AnyEvent::Curl::Multi. (AnyEvent::Net::Curl::Queued is actually "cheating", as it's non-lazy initialization is slower than the whole download cycle for everything else except WWW::Mechanize and LWP::UserAgent!!!)
URLs/s W::M LWP AE::C::M lftp H::T H::L YADA P::D curl L::C wget
WWW::Mechanize 195 -- -50% -60% -76% -76% -77% -81% -84% -92% -93% -94%
LWP::UserAgent 392 101% -- -21% -51% -52% -53% -61% -67% -83% -87% -88%
AnyEvent::Curl::Multi 493 153% 26% -- -39% -40% -41% -51% -59% -79% -83% -84%
lftp 802 312% 105% 63% -- -3% -4% -21% -33% -66% -73% -75%
HTTP::Tiny 825 323% 110% 67% 3% -- -1% -19% -32% -65% -72% -74%
HTTP::Lite 833 329% 113% 69% 4% 1% -- -17% -31% -65% -72% -74%
YADA 1013 419% 158% 105% 26% 23% 21% -- -16% -57% -66% -68%
Parallel::Downloader 1205 518% 207% 144% 50% 46% 44% 19% -- -49% -59% -62%
curl 2370 1115% 504% 380% 195% 187% 183% 134% 97% -- -20% -25%
LWP::Curl 2969 1421% 657% 501% 269% 260% 255% 193% 146% 25% -- -6%
wget 3166 1523% 707% 541% 294% 284% 279% 213% 163% 34% 7% --Kudos to Parallel::Downloader and LWP::Curl :)
ATTRIBUTES
allow_dups
Allow duplicate requests (default: false). By default, requests to the same URL (more precisely, requests with the same signature are issued only once. To seed POST parameters, you must extend the AnyEvent::Net::Curl::Queued::Easy class. Setting allow_dups to true value disables request checks.
completed
Count completed requests.
cv
AnyEvent condition variable. Initialized automatically, unless you specify your own. Also reset automatically after "wait", so keep your own reference if you really need it!
max
Maximum number of parallel connections (default: 4; minimum value: 1).
multi
Net::Curl::Multi instance.
queue
ArrayRef to the queue. Has the following helper methods:
queue_push: append item at the end of the queue;
queue_unshift: prepend item at the top of the queue;
dequeue: shift item from the top of the queue;
count: number of items in queue.
share
Net::Curl::Share instance.
stats
AnyEvent::Net::Curl::Queued::Stats instance.
timeout
Timeout (default: 60 seconds).
unique
Signature cache.
watchdog
The last resort against the non-deterministic chaos of evil lurking sockets.
METHODS
start()
Populate empty request slots with workers from the queue.
empty()
Check if there are active requests or requests in queue.
add($worker)
Activate a worker.
append($worker)
Put the worker (instance of AnyEvent::Net::Curl::Queued::Easy) at the end of the queue. For lazy initialization, wrap the worker in a sub { ... }, the same way you do with the Moose default => sub { ... }:
$queue->append(sub {
AnyEvent::Net::Curl::Queued::Easy->new({ initial_url => 'http://.../' })
});prepend($worker)
Put the worker (instance of AnyEvent::Net::Curl::Queued::Easy) at the beginning of the queue. For lazy initialization, wrap the worker in a sub { ... }, the same way you do with the Moose default => sub { ... }:
$queue->prepend(sub {
AnyEvent::Net::Curl::Queued::Easy->new({ initial_url => 'http://.../' })
});wait()
Process queue.
CAVEAT
The "Attempt to free unreferenced scalar: SV 0xdeadbeef during global destruction." message on finalization is mostly harmless.
SEE ALSO
AUTHOR
Stanislaw Pusep <stas@sysd.org>
COPYRIGHT AND LICENSE
This software is copyright (c) 2012 by Stanislaw Pusep.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.