/ 
WWW-Crawler-Mojo-0.14
River stage one • 1 direct dependent • 1 total dependent
/ WWW::Crawler::Mojo::Queue::Memory

NAME

WWW::Crawler::Mojo::Queue::Memory - Crawler queue with memory

SYNOPSIS

DESCRIPTION

Crawler queue with memory.

ATTRIBUTES

This class inherits all methods from WWW::Crawler::Mojo::Queue and implements following new ones.

redundancy

An subroutine reference called on enqueue process for avoiding redundant requests. It marks the job 'done' and returns 0, and next time returns 1.

if (!$queue->redundancy->($job)) {
    $queue->enqueue($job);
}

Defaults to a code that uses "no cleanup" storage. By replacing this you can control the memory usage.

$queue->redundancy(sub {
    my $d = $_[0]->digest;
    return 1 if $your_storage{$d};
    $your_storage{$d} = 1;
    return;
});

jobs

jobs.

METHODS

This class inherits all methods from WWW::Crawler::Mojo::Queue class and implements following new ones.

dequeue

Implementation for WWW::Crawler::Mojo::Queue interface.

enqueue

Implementation for WWW::Crawler::Mojo::Queue interface.

length

Implementation for WWW::Crawler::Mojo::Queue interface.

next

Implementation for WWW::Crawler::Mojo::Queue interface.

requeue

Implementation for WWW::Crawler::Mojo::Queue interface.

shuffle

Implementation for WWW::Crawler::Mojo::Queue interface.

AUTHOR

Keita Sugama, <sugama@jamadam.com>

COPYRIGHT AND LICENSE

Copyright (C) Keita Sugama.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.