Changes for version 0.09 - 2018-11-27
- Correct shebang lines in scripts
Documentation
how to install and setup TaskPipe
An overview of the TaskPipe framework
script to execute the TaskPipe Tool
Modules
A task management framework for building web scrapers and crawlers
handles file installs for TaskPipe
Parameter value interpolator for TaskPipe
handling matching of parameter values in the plan
Match handler for the $id parameter variable
match handler for the $name parameter variable
match handler for the $this parameter variable
module determining the parts of a plan parameter variable
generic class for TaskPipe Iterators
manages TaskPipe jobs
settings for the TaskPipe::JobManager module
Logging manager for TaskPipe
settings for the TaskPipe::LoggerManager module
open proxy management for TaskPipe
the base class for IPLists
Settings for TaskPipe::OpenProxyManager::IPList
PremProxy IP List
settings for TaskPipe::OpenProxyManager::IPList_PremProxy
ProxyNova IP List
Settings for TaskPipe::OpenProxyManager::IPList_ProxyNova
Xroxy IP List
settings for TaskPipe::OpenProxyManager::IPList_Xroxy
settings for TaskPipe::OpenProxyManager
Path settings for TaskPipe
TaskPipe global path settings
project path settings for TaskPipe
manage plan files for TaskPipe
settings for TaskPipe::Plan
read POD for TaskPipe
Settings for TaskPipe::PodReader
manage ports for TaskPipe
iterate mode type constraint
output mode type constraint
plan mode type constraint
scope mode type constraint
shell mode type constraint
array ref represented as a string type constraint
user agent type type constraint
information about the run (of the current plan)
base class for sample projects
sample project to scrape quotes for S&P500 companies
the default sample that is used when deploying files/db tables
manage schema connections for TaskPipe
settings for TaskPipe::SchemaManager
global schema settings for TaskPipe
project schema settings for TaskPipe
Global Schema Template for TaskPipe
schema template for the global daemon table
global schema template for job source
global schema template for open proxy source
Global Schema Template file for port source
global schema template for spawned source
global schema template for thread source
Project Schema Template for TaskPipe
schema template for the project error table
schema template for the project pinterp table
schema template for the project result table
schema template for project thread table
schema template for project xbranch_error table
schema template for project xbranch table
schema template for project xresult table
schema template for the SP500 sample project
schema template for the SP500 project company table
the class which all TaskPipe tasks should inherit from
map task names to modules
Settings for TaskPipe::Task
test settings for TaskPipe::Task
settings for the TaskPipe::TaskUtils module
record a record to the database
record a result set as a series of name/value pairs
record a result set as a series of name/value pairs
Base TaskPipe class for scraping a webpage
settings for TaskPipe::Task_Scrape
use a database table as the data source
use a file as a data source
the base class for file templates
template for the global config file
template for the project config
the template package for the 'system' config file
the base class for plan templates
template for the SP500 sample project plan
template for the default plan
base class for task templates
template for the Scrape_Companies task in the SP500 sample project
template for the Scrape_Quote task in the SP500 sample project
template for the Scrape_Stub task
manages threads for TaskPipe
Settings for TaskPipe::ThreadManager
the base class for the TaskPipe command line tool
base class for TaskPipe Tool commands
command to clear tables
command to deploy TaskPipe project files
command to deploy taskpipe tables
command to generate taskpipe schema files
command to run a TaskPipe plan
the TaskPipe setup command
show currently running TaskPipe jobs
command to stop a running taskpipe job
command to test an individual TaskPipe task
base calss for open proxy related commands
command to fetch open proxies
command to test open proxies
handles command line options for TaskPipe Tool
manage TOR processes for TaskPipe
settings for TaskPipe::TorManager
base class for managing requests in TaskPipe
settings for TaskPipe::UserAgentManger
standard useragent handler
useragent handler for phantomjs
base class for proxying useragents
useragent manager for making requests through an open proxy network
useragent manager for making requests through TOR
Provides
in lib/TaskPipe/Iterator_Array.pm
in lib/TaskPipe/Sample_Test.pm
in lib/TaskPipe/SchemaTemplate_Project_Test.pm
in lib/TaskPipe/SchemaTemplate_Project_Test/Result/City.pm
in lib/TaskPipe/SchemaTemplate_Project_Test/Result/Company.pm
in lib/TaskPipe/SchemaTemplate_Project_Test/Result/Employee.pm
in lib/TaskPipe/SchemaTemplate_Project_Test/Result/Operation.pm
in lib/TaskPipe/TaskUtils.pm
in lib/TaskPipe/Task_SeparateArray.pm
in lib/TaskPipe/Task_WhoIs/Settings.pm
in lib/TaskPipe/Template_Config_Project_SP500.pm
in lib/TaskPipe/WhoisManager/Settings.pm