NAME

TaskPipe::Task_Scrape - Base TaskPipe class for scraping a webpage

DESCRIPTION

This is the standard building block for creating a webpage-scraping task. To do this inherit from Task::Scrape using the following package format:

   package TaskPipe::Task_Scrape_MyScraper;

   use Moose;
   use Web::Scraper;
   extends 'TaskPipe::Task_Scrape';

   has test_pinterp => (is => 'ro', isa => 'ArrayRef[HashRef], default => sub{[

       {
           url => 'https://www.example.com/some-test-url',
           headers => {
               Referer => 'https://www.example.com/some-referer-url'
           }
       }
   
   ]});


   has ws => (is => 'ro', isa => 'Web::Scraper', default => sub{
       scraper {
           process 'div.some-class', 'results' => 'TEXT';
           result 'results'
       }
   });

   sub post_process {  # may or may not be necessary, depending
                       # on what is returned by ws

       my ($self,$results) = @_;

       # do something with the results returned from the web scraper

       return $results;
   }
   

test_pinterp allows you to specify test data which you can run the task against by typing

taskpipe test task --name=Scrape_MyScraper

at the command line.

It is assumed you want to use a Web::Scraper to scrape your page. If this is the case, just define a ws attribute as above. See the Web::Scraper manpage for more information on how to define a Web::Scraper.

Your task needs to return an arrayref of results (each result being a hashref). It's great if you can get ws to return this directly. Sometimes it is not possible to persuade your Web::Scraper to return results in this format. To make format corrections (remove records from the data etc) you can include a post_process subroutine. post_process receives the output from ws. Do what is needed, and make sure you return your results arrayref at the end.

AUTHOR

Tom Gracey <tomgracey@gmail.com>

COPYRIGHT AND LICENSE

Copyright (c) Tom Gracey 2018

TaskPipe is free software, licensed under

The GNU Public License Version 3