The Perl and Raku Conference 2025: Greenville, South Carolina - June 27-29 Learn more

 / 
Scrappy-0.94112090
River stage one • 1 direct dependent • 1 total dependent
/ Scrappy::Queue

NAME

Scrappy::Queue - Scrappy HTTP Request Flow-Control System

VERSION

version 0.94112090

SYNOPSIS


            

              
              
#!/usr/bin/perl
use Scrappy::Queue;
my  $queue = Scrappy::Queue->new;
    $queue->add($url);
     
    while (my $url = $queue->next) {
        ... $queue->add(...);
    }

DESCRIPTION

Scrappy::Queue provides a system for saving URLs to a recordset/queue and iterating of them using the Scrappy framework.

METHODS

list

The list method return the list of URLs in the queue. This is returned in list context.


            

              
              
my  $queue = Scrappy::Queue->new;
...
my  @list = $queue->list;

add

The add method adds new URLs to the queue. Duplicate URLs will be ignored.


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add($url);

clear

The clear method completely empties the queue and resets the cursor (loop position).


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
    $queue->clear;

reset

The reset method resets the cursor (loop position).


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
     
    while (my $url = $queue->next) {
        $queue->reset if ...; # beware the infinate loop
    }
     
    $queue->reset;

current

The current method returns the URL in the current loop position.


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
     
    while (my $url = $queue->next) {
        last if ...;
    }
     
    print 'great' if $url eq $queue->current;

next

The next method moves the cursor to the next loop position and returns the URL.


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
     
    while (my $url = $queue->next) {
        ...
    }

previous

The previous method moves the cursor to the previous loop position and returns the URL.


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
     
    while (my $url = $queue->next) {
        ...
    }
     
    print $queue->previous;

first

The first method moves the cursor to the first loop position and returns the URL.


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
     
    print $queue->first;

last

The last method moves the cursor to the last loop position and returns the URL.


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
     
    print $queue->last;

index

The index method moves the cursor to the specified loop position and returns the URL. The loop position is a standard array index position.


            

              
              
my  $queue = Scrappy::Queue->new;
    $queue->add(...);
    $queue->add(...);
    $queue->add(...);
     
    print $queue->index(1);

cursor

The cursor method returns the current loop position.


            

              
              
my  $queue = Scrappy::Queue->new;
    print $queue->cursor;

AUTHOR

Al Newkirk <awncorp@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2010 by awncorp.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.