Unit 2: Batch Job Submission

Exercises/Questions:
Short excursion on workspaces:
ws_allocate shortlived
ws_allocate alsoshortlived
ws_list # show all your workspaces and some info on them
ws_find shortlived # shows where it is
cd $(ws_find shortlived)
ls -l # well, it is empty
cd
ws_extend shortlived 90 # extend it to the maximum of 90 days
ws_extend shortlived 90
ws_extend shortlived 90
ws_extend shortlived 90 # end of possible extensions
ws_release shortlived # delete it
ws_release alsoshortlived
We load a module (math/matlab) and look at the example-jobscript provided.
module load math/matlab
cd $MATLAB_EXA_DIR
ls
less README
ws_allocate exercise 60 # 60 is max on bwuni, 90 on justus
ws_find exercise
cp * $(ws_find exercise)
cd $(ws_find exercise)

less README
less bwhpc_matlab_single.moab
EOT

We use a workspace for running the job inside it. Why may this actually be the wrong choice for bwhpc_matlab_single.moab?

click answer looking at the begin of the file yields:
########## Begin MOAB/Slurm header ##########
#
# Give job a reasonable name
#MOAB -N bwhpc_matlab_single
#
# Request number of nodes and CPU cores per node for job
#MOAB -l nodes=1:ppn=1
#
# Estimated wallclock time for job
#MOAB -l walltime=00:05:00
#
# Specify a queue class
look up the options -N and -l in the manpage to msub:
man msub
Search inside manpage for e.g. " -N" by typing "/ -N" (without the quotes)
Note that typing two spaces before "-N" will skip over any mentions of -N in the text describing different options.

Queues: Justus

See Queues at the bwhpc batch jobs page
Quick, short, normal and long queues exist, but you never have to select a queue manually. All your jobs will be put into the correct queue depending on the requested walltime. The limits for each queue still apply.

Queues: bwUniCluster

List of tables can be found at https://www.bwhpc-c5.de/wiki/index.php/Batch_Jobs_-_bwUniCluster_Features#msub_-q_queues

msub bwhpc_matlab_single.moab This returns a number, e.g. 6177199. This is your jobid.
Examples written here will use the jobid 6177199, use the real jobid of YOUR job instead.
showq # show all your jobs
showq -r # show only your jobs that are currently running
# do not forget to check the manpage!
man showq

checkjob 6177199 # moab summary of job
checkjob -v -v -v 6177199 # very verbose moab job summary


Inspect the job output file in the directory where you submitted the job.
This file contains everything that programs write to STDOUT while they run.

Submit the same job again, this time request 30GB of memory (RAM) for the job. click for answer

We will make an even simpler job that will run as long as we want and use a lot of CPU while it does.:
Create the file with: cat > load.moab << EOT
#!/bin/bash
module load mpi/openmpi

mpirun perl -e 'for(\$i=360,\$t=time;time-\$t<\$i;){}'
EOT
i=360 specifies 360 seconds of running time.
We specify all moab-options on the commandline, not inside the script: msub -l -l nodes=1:ppn=16,walltime=0:08:00 load.moab
BwUniCluster: You need to get an interactive node instead with:
msub -I -V -q develop -q singlenode -lnodes=1,ppn=16
you can then run the script with: bash load.moab &
Justus: You can ssh into any node that has a job of you running, because Justus will always reserve the whole node for you.
showq -r # wait for job to start
checkjob -v -v -v 6177199 | less
# find out which node the job runs on
ssh »nodename«
htop # should show 16 threads of CPU busy
ps aux | grep $USER
# if this doesn't find anything, username may be too long. Try
ps aux | grep ul_ # to find ulm users
vmstat 1 # end with ctrl-C
df -h
Can use gdb, strace etc on the PID of the job.
Repeat the same with nodes=1:ppn=8.

Why does htop show that only half of the cores on the node are in use while the other half is idle?

click for answer Run another job: echo sleep 10 | msub -l nodes=1:ppn=40 -l walltime 1:00:00 Find out why the job does not start.
Cancel the job.