NAME

Minion::Guide - An introduction to Minion

OVERVIEW

This document contains an introduction to Minion and explains the most important features it has to offer.

INTRODUCTION

Essentials every Minion developer should know.

Job queue

Job queues allow you to process time and/or computationally intensive tasks in background processes, outside of the request/response lifecycle of web applications. Among those tasks you'll commonly find image resizing, spam filtering, HTTP downloads, building tarballs, warming caches and basically everything else you can imagine that's not super fast.

Mojo::Server::Prefork                              +--------------+                     Minion::Worker
|- Mojo::Server::Daemon [1]       enqueue job ->   |              |   -> dequeue job    |- Minion::Job [1]
|- Mojo::Server::Daemon [2]                        |  PostgreSQL  |                     |- Minion::Job [2]
|- Mojo::Server::Daemon [3]   retrieve result <-   |              |   <- store result   |- Minion::Job [3]
+- Mojo::Server::Daemon [4]                        +--------------+                     |- Minion::Job [4]
                                                                                        +- Minion::Job [5]

They are not to be confused with time based job schedulers, such as cron or systemd timers, but Minion has built in support for cron style "Recurring jobs" when you want a task to run on a wall clock schedule.

Mojolicious

You can use Minion as a standalone job queue or integrate it into Mojolicious applications with the plugin Mojolicious::Plugin::Minion.

use Mojolicious::Lite -signatures;

plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};

# Slow task
app->minion->add_task(poke_mojo => sub ($job, @args) {
  $job->app->ua->get('mojolicious.org');
  $job->app->log->debug('We have poked mojolicious.org for a visitor');
});

# Perform job in a background worker process
get '/' => sub ($c) {
  $c->minion->enqueue('poke_mojo');
  $c->render(text => 'We will poke mojolicious.org for you soon.');
};

app->start;

Background worker processes are usually started with the command Minion::Command::minion::worker, which becomes automatically available when an application loads Mojolicious::Plugin::Minion.

$ ./myapp.pl minion worker

The worker process will fork a new process for every job that is being processed. This allows for resources such as memory to be returned to the operating system once a job is finished. Perl fork is very fast, so don't worry about the overhead.

Minion::Worker
|- Minion::Job [1]
|- Minion::Job [2]
+- ...

By default up to four jobs will be processed in parallel, but that can be changed with configuration options or on demand with signals.

$ ./myapp.pl minion worker -j 12

Jobs can be managed right from the command line with Minion::Command::minion::job, and recurring schedules with Minion::Command::minion::schedule.

$ ./myapp.pl minion job
$ ./myapp.pl minion schedule

You can also add an admin ui to your application by loading the plugin Mojolicious::Plugin::Minion::Admin. Just make sure to secure access before making your application publicly accessible.

# Make admin ui available under "/minion"
plugin 'Minion::Admin';

Deployment

To manage background worker processes with systemd, you can use a unit configuration file like this.

[Unit]
Description=My Mojolicious application workers
After=postgresql.service

[Service]
Type=simple
ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
KillMode=process

[Install]
WantedBy=multi-user.target

Consistency

Every new job starts out as inactive, then progresses to active when it is dequeued by a worker, and finally ends up as finished or failed, depending on its result. Every failed job can then be retried to progress back to the inactive state and start all over again.

                                                   +----------+
                                                   |          |
                                          +----->  | finished |
+----------+            +--------+        |        |          |
|          |            |        |        |        +----------+
| inactive |  ------->  | active |  ------+
|          |            |        |        |        +----------+
+----------+            +--------+        |        |          |
                                          +----->  |  failed  |  -----+
     ^                                             |          |       |
     |                                             +----------+       |
     |                                                                |
     +----------------------------------------------------------------+

The system is eventually consistent and will preserve job results for as long as you like, depending on "remove_after" in Minion. But be aware that failed results are preserved indefinitely, and need to be manually removed by an administrator if they are out of automatic retries.

While individual workers can fail in the middle of processing a job, the system will detect this and ensure that no job is left in an uncertain state, depending on "missing_after" in Minion. Jobs that do not get processed after a certain amount of time, depending on "stuck_after" in Minion, will be considered stuck and fail automatically. So an admin can take a look and resolve the issue.

FEATURES

Minion has many great features. This section is still very incomplete, but will be expanded over time.

Priorities

Every job enqueued with "enqueue" in Minion has a priority. Jobs with a higher priority get performed first, the default priority is 0. Priorities can be positive or negative, but should be in the range between 100 and -100.

# Default priority
$minion->enqueue('check_links', ['https://mojolicious.org']);

# High priority
$minion->enqueue('check_links', ['https://mojolicious.org'], {priority => 30});

# Low priority
$minion->enqueue('check_links', ['https://mojolicious.org'], {priority => -30});

You can use "retry" in Minion::Job to raise or lower the priority of a job.

$job->retry({priority => 50});

Job results

The result of a job has two parts. First there is its state, which can be finished for a successfully processed job, and failed for the opposite. And second there's a result data structure, that may be undef, a scalar, a hash reference, or an array reference. You can check both at any time in the life cycle of a job with "job" in Minion, all you need is the job id.

# Check job state
my $state = $minion->job($job_id)->info->{state};

# Get job result
my $result = $minion->job($job_id)->info->{result};

While the state will be assigned automatically by Minion, the result for finished jobs is usually assigned manually with "finish" in Minion::Job.

$minion->add_task(job_with_result => sub ($job) {
  sleep 5;
  $job->finish({message => 'This job should have taken about 5 seconds'});
});

For jobs that failed due to an exception, that exception will be assigned as result.

$minion->add_task(job_that_fails => sub ($job) {
  sleep 5;
  die 'This job should always fail after 5 seconds';
});

But jobs can also fail manually with "fail" in Minion::Job.

$minion->add_task(job_that_fails_with_result => sub ($job) {
  sleep 5;
  $job->fail({errors => ['This job should fail after 5 seconds']});
});

Retrieving job results is of course completely optional, and it is very common to have jobs where the result is unimportant.

Named queues

Each job can be enqueued with "enqueue" in Minion into arbitrarily named queues, independent of all their other properties. This is commonly used to have separate classes of workers, for example to ensure that free customers of your web service do not negatively affect your service level agreements with paying customers. The default named queue is default, but aside from that it has no special properties.

# Use "default" queue
$minion->enqueue('check_links', ['https://mojolicious.org']);

# Use custom "important" queue
$minion->enqueue('check_links', ['https://mojolicious.org'], {queue => 'important'});

For every named queue you can start as many workers as you like with the command Minion::Command::minion::worker. And each worker can process jobs from multiple named queues. So your workers can have overlapping responsibilities.

$ ./myapp.pl minion worker -q default -q important

There is one special named queue called minion_foreground that you should avoid using directly. It is reserved for debugging jobs with "foreground" in Minion.

Job progress

Progress information and other job metadata can be stored in notes at any time during the life cycle of a job with "note" in Minion::Job. The metadata can be arbitrary data structures constructed with scalars, hash references and array references.

$minion->add_task(job_with_progress => sub ($job) {
  sleep 1;
  $job->note(progress => '25%');
  sleep 1;
  $job->note(progress => '50%');
  sleep 1;
  $job->note(progress => '75%');
  sleep 1;
  $job->note(progress => '100%');
});

Notes, similar to job results, can be retrieved with "job" in Minion, all you need is the job id.

# Get job metadata
my $progress = $minion->job($job_id)->info->{notes}{progress};

You can also use notes to store arbitrary metadata with new jobs when you create them with "enqueue" in Minion.

# Create job with metadata
$minion->enqueue('job_with_progress', [], {notes => {progress => 0, something_else => [1, 2, 3]}});

The admin ui provided by Mojolicious::Plugin::Minion::Admin allows searching for jobs containing a certain note, so you can also use them to tag jobs.

Delayed jobs

The delay option of "enqueue" in Minion can be used to delay the processing of a job by a certain amount of seconds (from now).

# Job will not be processed for 60 seconds
$minion->enqueue('check_links', ['https://mojolicious.org'], {delay => 20});

You can use "retry" in Minion::Job to change the delay.

$job->retry({delay => 10});

Recurring jobs

For jobs that should run on a wall clock schedule, use Minion::Command::minion::schedule with a cron expression. Schedules are stored in the database as the source of truth, so they survive restarts and only fire once per interval no matter how many workers are running.

# Daily cleanup at 4am
$ ./myapp.pl minion schedule -e daily_cleanup -c '0 4 * * *' -t cleanup

A cron expression is five fields separated by spaces, one for each part of the wall clock. All times are interpreted in UTC, not local time.

* * * * *
| | | | |
| | | | +---- day of week  (0-6, 0=Sun; or SUN-SAT; 7 also means Sun)
| | | +------ month        (1-12; or JAN-DEC)
| | +-------- day of month (1-31)
| +---------- hour         (0-23)
+------------ minute       (0-59)

A * means "every value", so * * * * * reads as "every minute of every hour, every day". Pin a field to a specific number to fix that part of the clock.

0 4 * * *       # 04:00 every day
30 9 * * 1      # 09:30 every Monday
0 0 1 1 *       # midnight on January 1st (once a year)

Use /N for "every Nth value", a-b for ranges, and commas for lists.

*/15 * * * *    # every 15 minutes
0 9-17 * * *    # at the top of every hour from 09:00 to 17:00
0 9 * * 1-5     # 09:00 every weekday
0 0,12 * * *    # midnight and noon

Common patterns also have nicknames that expand to one of the forms above.

@hourly         # 0 * * * *
@daily          # 0 0 * * *
@midnight       # 0 0 * * *
@weekly         # 0 0 * * 0
@monthly        # 0 0 1 * *
@yearly         # 0 0 1 1 *

When both day-of-month and day-of-week are restricted, the job fires when either matches, following the standard Vixie cron(8) behavior. So 0 0 1 * 0 means "midnight on the 1st of the month and every Sunday at midnight". Times are interpreted in UTC.

Arguments, priorities, queues and retries can be set per schedule with the same options as Minion::Command::minion::job.

# Every five minutes, with arguments and high priority
$ ./myapp.pl minion schedule -e refresh -c '*/5 * * * *' -t refresh_cache -a '["users"]' -p 5

# Weekdays at 9, with retries
$ ./myapp.pl minion schedule -e weekday_report -c '0 9 * * 1-5' -t report -A 3

Adding a schedule with an existing name updates it; reusing the same cron expression preserves the next firing time, changing it recomputes it. Invalid expressions are rejected immediately.

# Pause and resume schedules without removing them
$ ./myapp.pl minion schedule -P daily_cleanup
$ ./myapp.pl minion schedule -r daily_cleanup

# Show or list schedules
$ ./myapp.pl minion schedule
$ ./myapp.pl minion schedule daily_cleanup

# Remove a schedule
$ ./myapp.pl minion schedule -R daily_cleanup

Schedules can also be declared in code with "schedule" in Minion and managed from the admin UI.

$minion->schedule(daily_cleanup => '0 4 * * *' => 'cleanup');

Running workers tick "dispatch_schedules" in Minion every dispatch_interval seconds (default 30) to enqueue any due jobs, coordinated through a Postgres advisory lock so a single dispatch cycle never fires the same schedule twice. If no worker has been running, the dispatcher will not catch up missed runs; the next firing time is always the next match strictly after the current time.

Expiring jobs

The expire option of "enqueue" in Minion can be used to limit for how many seconds (from now) a job should be valid before it expires and gets deleted from the queue.

# Job will vanish if it is not dequeued within 60 seconds
$minion->enqueue('check_links', ['https://mojolicious.org'], {expire => 60});

You can use "retry" in Minion::Job to reset the expiration time.

$job->retry({expire => 30});

Job dependencies

The parents option of "enqueue" in Minion can be used to make a new job depend on one or more existing jobs, which all need to have reached the state finished before it can be processed.

# Second job will not be processed until first job has finished
my $first = $minion->enqueue('step_one');
$minion->enqueue('step_two', [], {parents => [$first]});

This is commonly used to build workflows where tasks need to run in a specific order.

my $a = $minion->enqueue('prepare');
my $b = $minion->enqueue('process', [], {parents => [$a]});
my $c = $minion->enqueue('cleanup', [], {parents => [$b]});

If a parent job transitions to the failed state, all dependent jobs will remain inactive and wait until it has been retried successfully or removed. You can use the lax option to allow a job to be processed even if its parents have failed.

# Job will be processed even if parent jobs have failed
$minion->enqueue('cleanup', [], {parents => [$id], lax => 1});

Automatic retries

The attempts option of "enqueue" in Minion can be used to allow a job to be retried automatically if it fails, up to the specified number of attempts.

# Job will be attempted up to five times
$minion->enqueue('check_links', ['https://mojolicious.org'], {attempts => 5});

The delay between retries is determined by "backoff" in Minion, which defaults to (retries ** 4) + 15 (15, 16, 31, 96, 271, 640...) seconds, allowing roughly 25 attempts to be made over 21 days. You can change the formula by providing your own callback.

# Add some randomness to spread retry load
$minion->backoff(sub ($retries) { ($retries ** 4) + 15 + int(rand 30) });

Named locks

"lock" in Minion and "unlock" in Minion can be used to limit the concurrency of certain tasks, or to implement rate limiting for access to external services.

# Only one job should run at a time
$minion->add_task(do_unique_stuff => sub ($job, @args) {
  return $job->finish('Previous job is still active')
    unless $minion->lock('fragile_backend_service', 7200);
  ...
  $minion->unlock('fragile_backend_service');
});

The second argument is the number of seconds after which the lock will expire automatically, in case the job fails to release it. An optional limit argument can be used to allow more than one lock with the same name to be active at the same time.

# Only five jobs should run at a time and we try again later if necessary
$minion->add_task(do_concurrent_stuff => sub ($job, @args) {
  return $job->retry({delay => 30})
    unless $minion->lock('some_web_service', 60, {limit => 5});
  ...
  $minion->unlock('some_web_service');
});

For convenience you can also use "guard" in Minion to release the lock automatically as soon as the job is finished, even if it failed.

$minion->add_task(do_concurrent_stuff => sub ($job, @args) {
  return $job->retry({delay => 30})
    unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
  ...
});

Result promises

"result_p" in Minion can be used to wait for the result of a job asynchronously, without blocking the event loop. It returns a Mojo::Promise object that will be fulfilled when the job reaches the state finished, or rejected when it reaches the state failed.

# Enqueue job and receive the result at some point in the future
my $id = $minion->enqueue('foo');
$minion->result_p($id)->then(sub ($info) {
  say "Finished: $info->{result}";
})->catch(sub ($info) {
  say "Failed: $info->{result}";
})->wait;

Foreground debugging

"foreground" in Minion can be used to retry a job in the minion_foreground queue and perform it right away in the current process, making it very easy to debug a failed job with the full output of your application available. This is usually done from the command line with Minion::Command::minion::job.

$ ./myapp.pl minion job -f 10023

Worker signals

Worker processes can be controlled at runtime with signals. INT and TERM will stop the worker gracefully after finishing its current jobs, while QUIT will stop it immediately.

# Stop worker gracefully
$ kill -INT $worker_pid

Job processes spawned by the worker can also receive signals. INT and TERM start out with the operating system default, allowing jobs to install custom signal handlers to stop gracefully. USR1 and USR2 start out being ignored, and are available for jobs to use as they see fit.

$minion->add_task(myapp_task => sub ($job, @args) {
  local $SIG{INT} = sub { ...; exit };
  ...
});

Remote control

"broadcast" in Minion can be used to send remote control commands to one or more workers. Several commands are built in, such as stop to kill a job, kill to send a signal to a job process, and jobs to change how many jobs a worker runs concurrently. This is usually done from the command line with Minion::Command::minion::job.

# Kill job 10025
$ ./myapp.pl minion job -b stop -a '[10025]'

# Send USR1 signal to job 10026
$ ./myapp.pl minion job -b kill -a '["USR1", 10026]'

# Reduce concurrency of worker 23 to two jobs
$ ./myapp.pl minion job -b jobs -a '[2]' 23

You can also add your own commands to workers with "add_command" in Minion::Worker.

$worker->add_command(my_command => sub ($worker, @args) {
  ...
});

Testing

"perform_jobs" in Minion can be used to perform all jobs in the queue with a temporary worker, making it very easy to test background tasks without having to start a real worker process.

use Test::More;
use Mojolicious::Lite -signatures;

plugin Minion => {Pg => 'postgresql://postgres@/test'};

app->minion->add_task(add => sub ($job, $a, $b) {
  $job->finish($a + $b);
});

my $id = app->minion->enqueue(add => [1, 1]);
app->minion->perform_jobs;
is app->minion->job($id)->info->{result}, 2, 'right result';

done_testing;

Job iteration

"jobs" in Minion can be used to safely iterate through job information, with a variety of filters to narrow down the results.

# Iterate through all failed jobs for a task
my $jobs = $minion->jobs({states => ['failed'], tasks => ['check_links']});
while (my $info = $jobs->next) {
  say "$info->{id}: $info->{result}";
}

This is commonly used to build custom monitoring or to perform bulk operations on jobs.

# Retry all failed jobs in a queue
my $jobs = $minion->jobs({states => ['failed'], queues => ['important']});
while (my $info = $jobs->next) {
  $minion->job($info->{id})->retry;
}

Rate limiting

Named locks can also be used to limit how often a job accesses an external service, by letting them expire naturally instead of releasing them manually with "unlock" in Minion.

# Only a hundred jobs should run per hour
$minion->add_task(do_rate_limited_stuff => sub ($job, @args) {
  return $job->retry({delay => 3600})
    unless $minion->lock('some_web_service', 3600, {limit => 100});
  ...
});

Each lock acquired this way will count against the limit for the duration of its expiration time, effectively capping throughput to limit jobs per expiration period.

Custom workers

In cases where you don't want to use Minion together with Mojolicious, you can just skip the plugins and write your own worker scripts.

#!/usr/bin/perl
use strict;
use warnings;

use Minion;

# Connect to backend
my $minion = Minion->new(Pg => 'postgresql://postgres@/test');

# Add tasks
$minion->add_task(something_slow => sub ($job, @args) {
  sleep 5;
  say 'This is a background worker process.';
});

# Start a worker to perform up to 12 jobs concurrently
my $worker = $minion->worker;
$worker->status->{jobs} = 12;
$worker->run;

The method "run" in Minion::Worker contains all features you would expect from a Minion worker and can be easily configured with "status" in Minion::Worker. For even more customization options Minion::Worker also has a very rich low level API you could for example use to build workers that do not fork at all.

Task plugins

As your Mojolicious application grows, you can move tasks into application specific plugins.

package MyApp::Task::PokeMojo;
use Mojo::Base 'Mojolicious::Plugin', -signatures;

sub register ($self, $app, $config) {
  $app->minion->add_task(poke_mojo => sub ($job, @args) {
    $job->app->ua->get('mojolicious.org');
    $job->app->log->debug('We have poked mojolicious.org for a visitor');
  });
}

1;

Which are loaded like any other plugin from your application.

# Mojolicious
$app->plugin('MyApp::Task::PokeMojo');

# Mojolicious::Lite
plugin 'MyApp::Task::PokeMojo';

Task classes

For more flexibility, or if you are using Minion as a standalone job queue, you can also move tasks into dedicated classes. Allowing the use of Perl features such as inheritance and roles. But be aware that support for task classes is still EXPERIMENTAL and might change without warning!

package MyApp::Task::PokeMojo;
use Mojo::Base 'Minion::Job', -signatures;

sub run ($self, @args) {
  $self->app->ua->get('mojolicious.org');
  $self->app->log->debug('We have poked mojolicious.org for a visitor');
}

1;

Task classes are registered just like any other task with "add_task" in Minion and you can even register the same class with multiple names.

$minion->add_task(poke_mojo => 'MyApp::Task::PokeMojo');

You can continue with Mojolicious::Guides now or take a look at the Mojolicious wiki, which contains a lot more documentation and examples by many different authors.

SUPPORT

If you have any questions the documentation might not yet answer, don't hesitate to ask in the Forum or the official IRC channel #mojo on irc.libera.chat (chat now!).

To install Minion, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Minion

CPAN shell

perl -MCPAN -e shell
install Minion

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)