/ 
HPC-Runner-Slurm-2.33
River stage one • 1 direct dependent • 1 total dependent
/ HPC::Runner::Examples

Name

HPC::Runner::Examples

Version

This version describes HPC-Runner-Slurm 1.0

Description

The idea behind the Runner modules is to be able to run arbitrary bash with proper logging, catching STDOUT/ERROR and exit status, and when possible to run jobs in parallel with some job flow.

The modules are written with Moose, and can be overwritten and extended.

Logging is accomplished with Log::Log4perl.

Runner::Init is a base class thas has the variables common among HPC::Runner::Threads, HPC::Runner::MCE, and HPC::Runner::Slurm. All three modules have use a similar philosophy, but different technologies to implement it. For myself this was a woraround so I didn't have to learn to write MPI scripts, or have every tool be written into some sort of workflow manager.

HPC::Runner::Threads and HPC::Runner::MCE Example

An example infile would contain any command that can be executed from the command line. All the modules have a basic level of workflow management, meaning you can use the command 'wait' to wait for all other threads/processes to finish.

In the example directory there is a script called testioselect.pl. It is 100% from a thread on perlmonks discussing the proper use of IPC::Open3 found here. http://www.perlmonks.org/?node_id=151886. I based all the usage of running bash commands from the user abstract's post, only adding in the parts for logging.

You could create an test_threads/mce.in with the following.

perl example/testioselect.pl 1
perl example/testioselect.pl 2
perl example/testioselect.pl 3
perl example/testioselect.pl 4
#Wait for commands 1-4 to finish
wait
perl example/testioselect.pl 5
perl example/testioselect.pl 6
perl example/testioselect.pl 7
perl example/testioselect.pl 8
perl example/testioselect.pl 9
perl example/testioselect.pl 10
perl example/testioselect.pl 11
#Wait for commands 5-11 to finish
wait
perl example/testioselect.pl 12
perl example/testioselect.pl 13

And submit that to the HPC::Runner::MCE/Threads with the following.

mcerunner.pl --infile test_mce.in --outdir `pwd`/test --procs 4

Which would generate you the the test directory, and logs for the commands detailing STDOUT/STDERR, time and date, and run those commands 4 threads/processes at a time.

Each command gets its own log file, as well as a MAIN log file to detail how the job is running overall.

Trouble Shooting HPC::Runner::MCE/Threads

First of all, make sure your jobs run without the wrapper script. HPC::Runner::Threads/MCE only makes sure your threads start. It does not make sure your jobs exit successfully, but the exitcode will be in your log.

If you are using HPC::Runner::Threads your perl must be installed with thread capabilities.

HPC::Runner::Slurm Example

HPC::Runner::Slurm adds another layer to HPC::Runner::MCE or HPC::Runner::Threads by submitting jobs to the queing system Slurm. https://computing.llnl.gov/linux/slurm/. Slurm submits its jobs to different machines, or nodes, across a cluster. It is common for many users sharing the same space.

When I was first using slurm I wanted something that would automatically distribute my jobs across the cluster in a way that would get them done reasonably quickly. Most of the jobs being submitted were 'embarassingly parallel' and did not require much of the fine tuning slurm is capable of. For most jobs what we wanted to be able to do was take a list of jobs, chop them into pieces, take each piece and send it to a node, and then on that node run those jobs in parallel.

Here is a list of jobs in alljobs.in

job1
job2
job3
job4
wait
job5
job6
job7
job8

What I want is for Slurm to take 4 jobs at a time, submit those to a node. I don't want to do this all manually.

#!/bin/bash

#SBATCH --share
#SBATCH --get-user-env 
#SBATCH --job-name=alljobs_batch1
#SBATCH --output=batch1.out
#SBATCH --partition=bigpartition
#SBATCH --nodelist=node1_on_bigpartion

job1
job2
job3
job4

Ok, I don't really want that. I want all the logging, and since those jobs don't depend on one another I want to run them all in parallel. Because that is what HPC is all about. ;) So I run this command instead that uses the script that comes with HPC::Runner::Slurm.

slurmrunner.pl --infile alljobs.in --jobname alljobs --outdir alljobs

And have the following template files created and submitted to the queue.

Although it is not required to supply a jobname or an outdir, it is strongly recommended especially if you are submitting multiple jobs.

#!/bin/bash

#alljobs_batch1.sh

#SBATCH --share
#SBATCH --get-user-env 
#SBATCH --job-name=alljobs_batch1
#SBATCH --output=batch1.out
#SBATCH --partition=bigpartition
#SBATCH --nodelist=node1_on_bigpartion
#SBATCH --cpus-per-task=4

mcerunner.pl --infile batch1.in --procs 4

Where batch1.in contains our jobs1-4. The number that is in --cpus-per-task should be greater than or equal to the number of threads/processes that are running (procs). The default values in HPC::Runner::Slurm are fine, but if you change them make sure you stick with that rule.

This template and batch1.in is generated by the command and is submitted with the slurmjobid 123.

Then the next job batch is generated as alljobs_batch2.sh, and we tell slurm we want for it to be submitted after jobs1,2,3,4 exit successfully.

#!/bin/bash

#alljobs_batch2.sh

#SBATCH --share
#SBATCH --get-user-env 
#SBATCH --job-name=alljobs_batch2
#SBATCH --output=batch2.out
#SBATCH --partition=bigpartition
#SBATCH --nodelist=node2_on_bigpartion
#SBATCH --cpus-per-task=4
#Don't start this job until 123 submits successfully
#SBATCH --dependency=afterok:123

mcerunner.pl --infile batch2.in --procs 4

Customizing HPC::Runner::Slurm Input

More to come here. For now look at the documentation in HPC::Runner::Init and HPC::Runner::Slurm to see which variables are used.

Trouble Shooting HPC::Runner::Slurm

Make sure your paths are sourced correctly for slurm. The easiest way to do this is add all your paths to your ~/.bashrc, source it, and add the line

#SBATCH --get-user-env

to your submit script. By default this is already placed in the template, but if you decide to supply your own template you may want to add it.

If you are submitting a script that is not in your path, you probably want to give the full pathname for it, especially if supplying the outdir option. In general I think its always best to give the full pathname. If you are in the directory already and submitting from bash, just use backticks around pwd.

slurmrunner.pl --outdir `pwd`/jobout

Another common error is 'This node configuration is not available'. This could mean several things.

1. The node is down at the time of job submission
2. You are asking for more resources on a node than it has. If you ask for --cpus-per-task=32 and the node only has 16 cpus, you will get this error.
3. You misspelled the partition or nodename.

Number 2 will be improved upon in the next release so it queries slurm for the number of cpus available on a node at the time of submission. For now it must be manually set with --cpus-per-task