/ 
HPC-Runner-Scheduler-0.09
River stage one • 2 direct dependents • 2 total dependents
/ HPC::Runner::Usage

Name

HPC::Runner::Usage

HPC-Runner-Scheduler

Overview

The HPC::Runner modules are wrappers around running Gnu Parallel, Parallel::ForkManager, MCE/MCE::Queue, and job submission to a Slurm or PBS queue

Submit your commands

##Submit a job to slurm to spread across nodes
slurmrunner.pl --infile /path/to/fileofcommands --outdir slurmoutput --jobname slurmjob 

##Run in parallel using threads on a single noce
parallelrunner.pl --infile /path/to/fileofcommands --outdir threadsoutput --procs 4 

##Run in parallel using MCE on a single node
mcerunner.pl --infile /path/to/fileofcommands --outdir threadsoutput --procs 4

Run Your Command

The idea behind the HPC::Runner modules is to be able to run arbitrary bash with proper logging, catching STDOUT/ERROR and exit status, and when possible to run jobs in parallel with some job flow.

The modules are written with Moose, and can be overwritten and extended.

Logging is done with Log::Log4perl.

HPC::Runner is a base class thas has the variables common among HPC::Runner::Threads, HPC::Runner::MCE, and HPC::Runner::Slurm. All three modules have use a similar philosophy, but different technologies to implement it. For myself this was a workaround so I didn't have to learn to write MPI scripts, or have every tool be written into some sort of workflow manager.

The different runners each come with an executable script that should be installed in your path: mcerunner.pl, parallelrunner.pl, and slurmrunner.pl.

An Indepth Look

Single Node Execution

If you only have a single node to execute on, but still have many threads/processes available, you can use HPC::Runner::MCE or HPC::Runner:Threads. Both have job logging and workflow control.

Example for Runner::Threads and Runner::MCE

An example infile would contain any command that can be executed from the command line. All the modules have a basic level of workflow management, meaning you can use the command 'wait' to wait for all other threads/processes to finish.

In the example directory there is a script called testioselect.pl. It is 100% from a thread on perlmonks discussing the proper use of IPC::Open3 found here. http://www.perlmonks.org/?node_id=151886. I based all the usage of running bash commands from the user abstract's post, only adding in the parts for logging.

You could create a test_threads/mce.in with the following.

test_threads/mce.in

It is ALWAYS a good idea to put full paths when running arbitrary scripts. Jobs will always be run from the your current working directory, but its still a good idea!

And submit that to the Runner::MCE/Threads with the following.

#Using MCE

mcerunner.pl --infile test_mce.in --outdir `pwd`/test --procs 4

# OR
# Using Parallel::Forkmanager

parallelrunner.pl --infile test_mce.in --outdir `pwd`/test --procs 4

Which would generate you the the test directory, and logs for the commands detailing STDOUT/STDERR, time and date, and run those commands 4 threads/processes at a time.

Each command gets its own log file, as well as a MAIN log file to detail how the job is running overall.

Trouble Shooting mcerunner and parallelrunner

First of all, make sure your jobs run without the wrapper script. Runner::Threads/MCE only makes sure your threads/processes start. It does not make sure your jobs exit successfully, but the exitcode will be in your log.

View your logs in outdir/prunnerrunnerdatetimerandomstr/CMD#datetime. This will give you the STDOUT/STDERR.

Full Path Names

Please give all your commands, infiles, and outdirectories the full path names. If you are executing arbitrary script you should give either the full path name or the path should be in your ENV{PATH}. HPC::Runner::Init will do some guessing on the infile and outdir parameters using File::Spec, but this is no guarantee!

If you are using Runner::Threads your perl must be installed with thread capabilities.

MultiNode Job Submission

This documentation was initially written for Slurm, but for the most part the concepts and requirements are the same across schedulers (Slurm, PBS, SGE, etc).

HPC::Runner::Slurm Example

HPC::Runner::Slurm adds another layer to HPC::Runner::MCE or HPC::Runner::Threads by submitting jobs to the queing system Slurm. https://computing.llnl.gov/linux/slurm/. Slurm submits its jobs to different machines, or nodes, across a cluster. It is common for many users sharing the same space.

When I was first using slurm I wanted something that would automatically distribute my jobs across the cluster in a way that would get them done reasonably quickly. Most of the jobs being submitted were 'embarassingly parallel' and did not require much of the fine tuning slurm is capable of. For most jobs what we wanted to be able to do was take a list of jobs, chop them into pieces, take each piece and send it to a node, and then on that node run those jobs in parallel.

alljobs.in

job1
job2
job3
job4
# Lets tell mcerunner/parallelrunner/slurmrunner.pl to execute jobs5-8 AFTER jobs1-4 have completed
# wait
job5
job6
job7
job8

What I want is for Slurm to take 4 jobs at a time, submit those to a node. I don't want to do this all manually for obvious reasons.

Slurm Template

#!/bin/bash
#SBATCH --share
#SBATCH --get-user-env
#SBATCH --job-name=alljobs_batch1
#SBATCH --output=batch1.out
#SBATCH --partition=bigpartition
#SBATCH --nodelist=node1onbigpartion

#Here are the jobs!
job1
job2
job3
job4

Ok, I don't really want that. I want all the logging, and since those jobs don't depend on one another I want to run them all in parallel. Because that is what HPC is all about. ;) So I run this command instead that uses the script that comes with Runner::Slurm.

slurmrunner.pl --infile pwd/alljobs.in --jobname alljobs --outdir pwd/alljobs

And have the following template files created and submitted to the queue.

Although it is not required to supply a jobname or an outdir, it is strongly recommended especially if you are submitting multiple jobs.

Slurm Template with Batched Job

#!/bin/bash
##alljobs_batch1.sh
#SBATCH --share
#SBATCH --get-user-env
#SBATCH --job-name=alljobs_batch1
#SBATCH --output=batch1.out
#SBATCH --partition=bigpartition
#SBATCH --nodelist=node1onbigpartion
#SBATCH --cpus-per-task=4
# 
# #Take out jobs, batch them out to a node, and run them in parallel
# mcerunner.pl --infile batch1.in --procs 4 --outdir /outdir/we/set/in/slurmrunner.pl

Where batch1.in contains our jobs1-4. The number that is in --cpus-per-task should be greater than or equal to the maximum number of threads/processes that are run in parallel (procs). The default values in HPC::Runner::Slurm are fine, but if you change them make sure you stick with that rule.

This template and batch1.in is generated by the command and is submitted with the slurmjobid 123.

Then the next job batch is generated as alljobs_batch2.sh, and we tell slurm we want for it to be submitted after jobs1,2,3,4 exit successfully.

Slurm Template with Dependency

#!/bin/bash
##alljobs_batch2.sh
#SBATCH --share
#SBATCH --get-user-env
#SBATCH --job-name=alljobs_batch2
#SBATCH --output=batch2.out
#SBATCH --partition=bigpartition
#SBATCH --nodelist=node2onbigpartion
#SBATCH --cpus-per-task=4
# 
# #Don't start this job until 123 submits successfully
# SBATCH --dependency=afterok:123
#  
# mcerunner.pl --infile batch2.in --procs 4 --outdir /outdir/we/set/in/slurmrunner.pl

Customizing HPC::Runner::Slurm Input

Since the HPC::Runner modules are written in Moose, they can be overridden and extended in the usual fashion. Logging is done with Log4Perl, so any of the appenders can be used. The default is to log to files, but what if you want to log to rsyslog or a database?

Extend slurmrunner.pl to add your own custom loggin

#!/usr/bin/env perl
#slurmrunner_rsyslog.pl
 
package Main;
use Moose;
 
extends 'HPC::Runner::Slurm';
 
 
use Log::Log4perl;
 
#Lets override init_log with our own subroutine...
 
sub init_log {
    my $self = shift;
    my $filename = $self->logdir."/".$self->logfile;
my $log_conf =<<EOF;
############################################################
#  Log::Log4perl conf - Syslog                             #
############################################################
log4perl.rootLogger                = DEBUG, SYSLOG, FILE
log4perl.appender.SYSLOG           = Log::Dispatch::Syslog
log4perl.appender.SYSLOG.min_level = debug
log4perl.appender.SYSLOG.ident     = slurmrunner
log4perl.appender.SYSLOG.facility  = local1
log4perl.appender.SYSLOG.layout    = Log::Log4perl::Layout::SimpleLayout
log4perl.appender.FILE           = Log::Log4perl::Appender::File
log4perl.appender.FILE.filename  = $filename
log4perl.appender.FILE.mode      = append
log4perl.appender.FILE.layout    = Log::Log4perl::Layout::PatternLayout
log4perl.appender.FILE.layout.ConversionPattern = %d %p %m %n
EOF
    Log::Log4perl::init(\$log_conf);
    my $log = Log::Log4perl->get_logger();
    return $log;
};
 
Main->new_with_options->run;
 
1;

Trouble Shooting Runner::Slurm

Make sure your paths are sourced correctly for slurm. The easiest way to do this is add all your paths to your ~/.bashrc, source it, and add the line

#SBATCH --get-user-env>

to your submit script. By default this is already placed in the template, but if you decide to supply your own template you may want to add it.

If you are submitting a script that is not in your path, you probably want to give the full pathname for it, especially if supplying the outdir option. In general I think its always best to give the full pathname.

If you are in the directory already and submitting from bash, just use backticks around pwd.

Another common error is 'This node configuration is not available'. This could mean several things.

1. The node is down at the time of job submission
2. You are asking for more resources on a node than it has. If you ask for --cpus-per-task=32 and the node only has 16 cpus, you will get this error.
3. You misspelled the partition or nodename.

Point 2 will be improved upon in the next release so it queries slurm for the number of cpus available on a node at the time of submission. For now it must be manually set with --cpus-per-task

Authors and Contributors

Jillian Rowe in collaboration with the ITS Advanced Computing Team at Weill Cornell Medical College in Qatar.

Acknowledgements

This module was originally developed at and for Weill Cornell Medical College in Qatar. With approval from WCMC-Q, this information was generalized and put on github, for which the authors would like to express their gratitude. Also to all the HPC users at WCMCQ, who all gave their input.

The continued development of the HPC::Runner modules is supported by NYUAD, at the Center for Genomics and Systems Biology.