Slurm Batch Scripting

Hi everybody, this is Albert from the Center
for High Performance Computing at the University of Utah, and in this tutorial, we will be
talking about Slurm batch jobs and batch scripting. The first question is, when should you be
using the batch system? Well, you should use the batch system when
you have a large job that is running in parallel that is heavy-duty computation and anything
that is, really, high-performance computing. To use the batch system, you have to really
have a grasp on scripting. Now, there will be another scripting tutorial
that you can watch, but we’re going to go through the basics of what you need to put
into a batch script in order to run on the batch system here. What goes into a batch script? The first thing that you put into a batch
script under a Slurm system is a set of “SBATCH” directives. SBATCH directives tell the scheduler what
to use in terms of number of nodes, how many cores per node, how many tasks across all
of those nodes, the account and the partition that you want to use, how long you’re going
to run for, all of those pieces of information that are useful for telling the scheduler
what your job looks like. Once you have your SBATCH directives set up,
you need to set up your environment. At CHPC, what that would entail is loading
some modules, maybe setting up some additional PATH variables, so that you can run a specific
program that you wrote yourself; basically setting up your environment so that you can
run your job. Now, a lot of jobs will also need a scratch
directory to run from. Usually, it’s good practice to set up your
scratch directory in your batch script as well before you run that. In addition to this, often times you’ll want
to be copying your data to scratch. This is especially true of programs that have
heavy IO loads. If you anticipate having lots of writes, then
you should really be doing your writing in scratch. If you’re doing something smaller, more lightweight,
that isn’t going to impact the file servers that much, then you can run from your home
directory, but this is generally not recommended. Now, obviously, if you’re running a batch
script, you want to run your job, and you want to run it in parallel with MPI if you
need to. Obviously, you want to make sure that you
run your job in your batch script. Otherwise, you’re not really doing anything. Now, along with step number four here, you
want to make sure that if you have data that you’re writing to scratch, that you copy it
to a safe location. Now, this could be to your home directory,
this could be to your group space; either way, if you’re writing to scratch, make sure
you copy from scratch so you don’t lose data in scratch. The scratch file server at CHPC are scrubbed
after thirty days. Finally, if your program writes any temporary
files, or if you have leftover scratch data, then you should remove those so you can be
a good citizen. So, these are the things that you need to
put into a batch script in order to make it run well. Now I’m going to show you an example script—a
very simple one that does some actual work for me—so if we do an “ls,” you can see
that we have this “run-qe.slurm” that you may have seen in our basic Slurm tutorial. So, I’m going to show you this script. Here we have our script. Now, the question is, what do you put into
this? Now, like I was saying before, you want to
put in a whole bunch of SBATCH directives. Before you do that, you want to make sure
you put in your shebang. So, here we can see that this is a bash script;
if you’re running a bash script, you want to be sure to use bash, while if you’re using
csh, you want to make sure to use tcsh. Either way, if you don’t put this in here,
then Slurm will not be able to determine how to run this script. It’ll only say, “Here’s this weird script
that I don’t know what to do with.” Now, the SBATCH directive for time is straightforward,
you just do “–time” equals the time that you are running. This is seconds, minutes, and hours. This directive shows the number of nodes;
this one determines the number of tasks. The number of tasks is spread out across however
many nodes you have, so if you have 16 tasks on one node, that’ll be 16 tasks per node. If you have 16 tasks but 2 nodes, then you’ll
have 8 tasks per node. Now, you can also tune this with different
parameters; for example, we have a parameter that is the number of tasks per node, and
you can also ask for different processors per node; those are all detailed on our Slurm
web page, but for the most part, using “ntasks” is straightforward. Here we have the account and partition, and
then here we have this “-o” flag, which tells the scheduler what the output name of your
standard output is. So, you can imagine that if this script had
errors in it, you would want to be writing this to standard output somehow so that you
could see those errors. Using the “-o” flag is how you do this. This right here is a name. You can write whatever name you want. I have this with the prefix “qe-” because
this is a Quantum Espresso job. The suffix here, the “%j,” is a special syntax
for Slurm, and it only works in Slurm, which will use the job number in place of something
else. If you wanted to append the job number to
your output name, then this is how you would do that. The next step is obviously to set up your
environment, and that’s what I do in this section of the script. I set up some convenience variables; I set
the name of my input file, which is actually just a prefix in this instance, I have my
data directory where my data is being stored, and then I have a scratch directory specified
because Quantum Espresso needs a scratch. Now, in addition to this, I also load the
correct modules in order to run this. Now, when you’re using modules, I recommend
that when you use batch scripts you start out by doing a “module purge.” This starts you out with a clean environment
and it makes things more predictable when you actually get there. This is because Slurm will carry over your
environment variables for your current environment. Then we do the “module load,” we do “intel,”
“impi,” “qe,” because that’s the order that we need, and this sets up the environment
that I want to be running in. Now, the variables set up here are totally
up to you, you don’t have to do this, but this is simply just convenience. Now, I do need to make sure that this directory
exists, so you can do this with a “mkdir” command, “-p” will make sure the directory
is created regardless of whether or not the parent directories exist, and obviously we
have the “$SCRATCH,” which refers back to this variable. Now, here we see that I have written this
comment “copy data to scratch here if you need to.” With Quantum Espresso, I don’t actually need
to copy my data to scratch, because I have a very limited amount of writing that I do
to my output. Most of it is handled by Quantum Espresso,
and this scratch directory which I specified is in the input file for Quantum Espresso,
so I don’t have to worry about it, but this is where you would perform that copy if you
needed to do so. Here we have my execution of the job, which
is simply an “mpirun.” Since I’ve loaded the “impi” module, mpirun
is already detected. Then I do the number of processes, which you
use the “$SLURM_NTASKS” variable to specify the number of tasks if you want to do this. For Slurm, you don’t actually need to put
this here, but I put it there for the sake of sanity, to make sure that I know what’s
going on, and then here we run Quantum Espresso, which is the name of the program, the input
file, and then the output file. Now, a good practice to adopt with your output
files is that if you’re writing an output file and you have control over what you can
name it, that you use the “$SLURM_JOB_ID” variable to “tag” that file with the job ID. This is useful to you so that you can keep
your input files dependent of each other and also so that when you are having trouble with
jobs you can easily find the job ID and tell us, the staff at CHPC, so we can help you. Now, you have an alternative to running “mpirun,”
and it can be somewhat complicated to run, but you can just use “srun.” Slurm is aware of MPI distributions and the
ones we use at CHPC will allow you to use srun in lieu of mpirun. There are some things that change between
versions of MPI, but you can use srun if you so desire. Now, the final step that you want to make
sure that you do if you’ve copied to scratch is to copy data from scratch. You can copy to your home directory, group
space, or otherwise. You just have to make sure that you have enough
space to make that copy. Then, finally, the last step is to clean up
your scratch directories and other temporary files. In this case, I want to clean up my scratch
directory, because I don’t need to keep the stuff there. You can clean up the environment and make
sure to free up space on scratch areas for other users. Now I’m going to quickly show you an example
of this. In my directory, I have my “run-que.slurm,”
here’s my input file. If I do “sbatch run-qe.slurm,” then I submit
my job. If I do an “squeue,” you can see that my job
is now running. Now, I’m expecting two output files here. One of them is the standard output and the
other is the output. You’ll notice that even though my job is running,
I’m not seeing this. This is because there’s a slight delay between
the filesystems being written to and then updating on the client. If you don’t immediately see output files,
it’s because the file servers are just taking a little time to make sure that you can see
those. And here we can see that I actually have my
two output files; here’s the output that I expect to see for my program, which is just
the useful output; if I look at it really quickly, you can see that it’s run and that
the job is done (it didn’t take very long to run—only about 14 seconds, which is expected). We also have the standard output file from
the script. Now, whether or not you see anything in that
output file is up to your program. You can see that there is no output, which
means my script worked right, and there were no errors, but it just depends on what your
script looks like. With that, that’s everything I have to show
you. I hope you enjoyed this tutorial and that
you learned something. If you have questions, you can email us at
[email protected] or visit us in our offices in INSCC 405. Thanks for watching!

4 thoughts on “Slurm Batch Scripting

  1. I'm an auditory learner, and it was very helpful hearing someone explain it instead of reading the man page 🙂

  2. Thank you for the clear presentation!! I am starting to study SLURM for computer vision & machine learning implementation. Could you give an example on how to train ConvNet architecture (Keras-based model) by using SLURM?

Leave a Reply

Your email address will not be published. Required fields are marked *