/ 
App-ppll-v0.0.1
River stage zero No dependents
/ ppll

NAME

ppll

SYNOPSIS

ppll [OPTIONS] COMMAND [ARGS]
ppll [OPTIONS] -c COMMAND [PARAMETERS]
ppll --help
ppll --version

In the first form, all arguments after COMMAND are considered arguments to COMMAND, not options to ppll, even if they begin with a hyphen.

DESCRIPTION

ppll is a tool to control the execution of commands. It can run commands in parallel, construct commands from lists of parameters, and more.

It handles the output of commands and can prefix lines with which command produced it, print timestamps, etc.

ppll has functionality similar to xargs and parallel.

PARAMETERS AND JOBS

‘Parameters’ is what makes ppll run multiple commands. In one way or another ppll can be given a number of parameters, and it will create and run one ‘job’ for each parameter.

If no parameter-related option is used ppll will read the parameters from STDIN (if it’s not a TTY). Ordinarily, each line read becomes one parameter, but there are options to control this.

The other important input to ppll is the ‘command’. The command-line arguments after all options make up the command and its arguments.

Each job will execute the command. All occurrences of {} in any part of the command (and its arguments) will be replaced with the job’s parameter. If there are no such occurrences the parameter will be appended as an additional argument.

For example:

ppll echo {} <<EOF
cow
pig
sheep
EOF

Will create jobs that execute echo cow, echo pig and so on. An other way to do the same thing would be:

ppll echo <<EOF
cow
pig
sheep
EOF

{} becomes useful when it’s not supposed to be the last argument, e.g:

ppll cp {} farm/ <<EOF
cow
pig
sheep
EOF

Or when it’s supposed to be just a part of an argument, or show up more than once, e.g:

ppll curl -o photo-of-a-{}.jpg http://example.com/img/{}.jpg <<EOF
cow
pig
sheep
EOF

The above would run curl -o photo-of-a-cow.jpg http://example.com/img/cow.jpg, etc.

JOB CONTROL

The default is to run several such jobs in parallel. Once a job is finished another one is started, until there are no more jobs to run.

The number of jobs run at the same time is controlled by the --jobs option. The default is one per CPU (but not necessarily one on each CPU).

If a job fails (exits with a code other than 0), no more jobs will be started. ppll attempts to stop other running jobs.

The --force option tells ppll to not care about failed jobs and always keep starting new ones. Any failed jobs are still noted and will still affect ppll’s own exit code.

SERIAL LINES, PARALLEL FIELDS

This feature deserves its own section.

It can be used to parallelise heavy tasks, while also making sure certain tasks are run in a particular order.

Each line of input will create a ‘group’ of parameters, that will be run before the next group starts. Each line is split up into individual parameters (using the IFS environment variable), that will be run at the same time.

An example:

ppll --slpf wake-up.sh <<EOF
rooster
hens chickens
cows pigs sheep
cat
EOF

The above would run wake-up.sh first for rooster. Then it would run it for hens and chickens, at the same time, but after the rooster job has completed. Then cows, pigs and sheep in parallel, and so on.

OPTIONS

GENERAL

--help
-h
-?

Print (this) help and exit.

--version
-V

Print ppll’s version and exit.

PARAMETERS AND JOBS

--command COMMAND
-C COMMAND

Run COMMAND as a shell command with each parameter passed as an argument. This allows for shell constructs, such as piping, redirecting, etc.

If used then the rest of the command-line arguments are used as parameters, instead of a command.

More or less the same as:

ppll $SHELL -c <COMMAND> -

($SHELL defaults to /bin/sh.)

--delimiter DELIMITER
-d DELIMITER

Use DELIMITER, as a regex, to split lines into fields. Implies --fields.

--empty

Include empty parameters. These are otherwise discarded.

--fields
-f

Alias for --mode fields.

--force
-k

Keep starting new jobs even if a job has failed.

--jobs N
-j N

Run (up to) this many jobs in parallel.

--lines
-l

Alias for --mode lines.

--mode MODE

How input should be turned into parameters. MODE is one of:

auto (default)

If there’s only one line of input, behave as if fields, otherwise as if lines.

fields

Split lines into fields and use each field as a parameter. Splitting is done using a delimiter set with --delimiter, or the IFS environment variable.

lines

Use each line as a parameter.

slpf

Split each line into parameters. All parameters from a single line will be run in parallel, but lines will be run serially.

See also: --fields, --lines, --serial-lines-parallel-fields

--random
-R

Shuffle the parameters.

--replace-string STRING
-replstr STRING
-I=s STRING

Use STRING instead of {} as a placeholder for the parameter.

--reverse
-r

Use the parameters in reverse order.

--sequence SPEC
-s SPEC

Use SPEC as parameters.

SPEC is in one of the forms:

N..M

Parameters are from N to M, inclusive.

M

Parameters are from 1 to M, inclusive.

--serial-lines-parallel-fields
-slpf

Alias for --mode slpf.

OUTPUT

--colour
--color
--no-colour
--no-color

Turn on or off ANSI colours in the output.

The default is to enable colours if STDOUT is a TTY.

--markers
--no-markers

Prefix (the default), or don’t prefix, jobs’ STDOUT and STDERR output with > or ≫, respectively.

--quiet
-q

Print less. Can be repeated for less and less.

--summary
-S
--no-summary

Print (the default if there’s more than one parameter), or don’t print, a summary of how many jobs ran and which of them failed.

--timestamp-format FORMAT

Format timestamps using FORMAT. See DateTime::strftime.

The default is %T.%3N.

Implies --timestamps.

--timestamps
-t

Prefix lines from jobs’ output with a timestamp. The time printed will be the approximate time the line was printed by the job.

--verbose
-v

Print more. Can be repeated for more and more.

DIAGNOSTICS

CONFIGURATION AND ENVIRONMENT

ppll uses some environment variables in some cases:

COLUMNS

If set then used when printing things across the whole width of the terminal. If not set then ppll will try to work out the terminal width in other ways, falling back to 80.

IFS

Used for splitting lines into fields, e.g. with --fields or --serial-lines-parallel-fields.

SHELL

Used with --commmand (-c).

EXAMPLES

ppll -t make

Run a long-running command, e.g. make, and prefix the output with timestamps.

ppll -j 10 -s 1..1000 gobbledygook --with-rigmarole --number

For each number from 1 to 1,000 (inclusive), run the gobbledygook command with some arguments. Add the a number as the last argument, e.g. gobbledygook --with-rigmarole --number 42.

Run no more than 10 such commands at a time.

ppll curl -o photo-of-a-{}.jpg http://example.com/img/{}.jpg <<EOF
cow
pig
sheep
EOF

Download three files in parallel. Note the use of {} to use each parameter in two places in each command.

SEE ALSO

xargs

POSIX, and thus available almost everywhere, and can run multiple commands based on a list of parameters, much like ppll.

GNU Parallel

Also runs commands in parallel, but with an interface quite different to ppll’s.

moreutil’s parallel and ts

moreutils contains a parallel tool similar yet different to GNU Parallel.

It also has a separate tool for adding timestamps to output.

AUTHOR

Theo -q Willows, <theo@willows.se>

BUGS AND LIMITATIONS

Please report any bugs or feature requests through the web interface at https://gitlab.com/munkei-software/ppll/issues.

SUPPORT

You can find documentation for this module with the perldoc command.

perldoc ppll

Or:

ppll --help

You can also look for information at:

LICENSE AND COPYRIGHT

Copyright 2019 Theo Willows.

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.