NAME
Mojo::UserAgent::Role::Queued - A role to process non-blocking requests in a rate-limiting queue.
SYNOPSIS
my
$ua
= Mojo::UserAgent->new->with_roles(
'+Queued'
);
$ua
->max_redirects(3);
$ua
->max_active(5);
# process up to 5 requests at a time
for
my
$url
(
@big_list_of_urls
) {
$ua
->get(
$url
,
sub
{
my
(
$ua
,
$tx
) =
@_
;
if
(
$tx
->success) {
say
"Page at $url is titled: "
,
$tx
->res->dom->at(
'title'
)->text;
}
});
};
# works with promises, too:
my
@p
=
map
{
$ua
->get_p(
$_
)->then(
sub
{
pop
->res->dom->at(
'title'
)->text })
->
catch
(
sub
{
say
"Error: "
,
@_
})
}
@big_list_of_urls
;
Mojo::Promise->all(
@p
)->
wait
;
DESCRIPTION
Mojo::UserAgent::Role::Queued manages all non-blocking requests made through Mojo::UserAgent in a queue to limit the number of simultaneous requests.
Mojo::UserAgent can make multiple concurrent non-blocking HTTP requests using Mojo's event loop, but because there is only a single process handling all of them, you must take care to limit the number of simultaneous requests you make.
Some discussion of this issue is available here http://blogs.perl.org/users/stas/2013/01/web-scraping-with-modern-perl-part-1.html and in Joel Berger's answer here: http://stackoverflow.com/questions/15152633/perl-mojo-and-json-for-simultaneous-requests.
Mojo::UserAgent::Role::Queued tries to generalize the practice of managing a large number of requests using a queue, by embedding the queue inside Mojo::UserAgent itself.
EVENTS
Mojo::UserAgent::Role::Queued adds the following event to those emitted by Mojo::UserAgent:
queue_empty
$ua
->on(
queue_empty
=>
sub
{
my
(
$ua
) =
@_
; .... })
Emitted when the queue has been emptied of all pending jobs. In previous releases, this event was called stop_queue
(this is a breaking change).
ATTRIBUTES
Mojo::UserAgent::Role::Queued has the following attributes:
max_active
$ua
->max_active(5);
# execute no more than 5 transactions at a time.
"Execute no more than "
,
$ua
->max_active,
" concurrent transactions"
Parameter controlling the maximum number of transactions that can be active at the same time.
LICENSE
Copyright (C) Dotan Dimet.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
AUTHOR
Dotan Dimet <dotan@corky.net>