What's Next?

You should now know the following:

1. Basic orginization of Biocluster
2. How to login to Biocluster
3. How to use the Module system to gain access to Biocluster software
4. CPU, storage, and memory limitations (quotas and hardware limits)

Managing Jobs

Submitting and managing jobs is at the heart of using the cluster.  A 'job' refers to the script, pipeline or experiment that you run on the nodes in the cluster.

Queues/Partitions

In the past we used queues under the old Torque system, we now refer to these logically grouped nodes as partitions. There are several different partitions available for cluster users to send jobs to:

• batch
  
• Nodes: c01-c48
• Cores: AMD, 256 per user
• RAM: 1 GB default
• Time (walltime): 168 hours (7 days) default
• highmem
  
• Nodes: h01-h04
• Cores: Intel, 32 per user
• RAM: 16 GB min and 1024 GB max
• Time (walltime): 48 hours (2 days) default
• gpu
  
• Nodes: gpu01-gpu02
• Cores: Intel, 16 per user
• RAM: 128 GB default
• Time (walltime): 100 hours  default
• intel
  
• Default partition
• Nodes: i01-i12
• Cores: Intel, 64 per user
• RAM: 1 GB default
• Time (walltime): 168 hours (7 days) default
• Group Partition
• This partition is unique to the group, if your lab has purchased nodes then you will have a priority partition with the same name as your group (ie. girkelab).

In order to submit a job to different partitions add the optional '-p' parameter with the name of the partition you want to use:

sbatch -p batch SBATCH_SCRIPT.sh

sbatch -p highmem SBATCH_SCRIPT.sh

sbatch -p gpu SBATCH_SCRIPT.sh

sbatch -p intel SBATCH_SCRIPT.sh

sbatch -p mygroup SBATCH_SCRIPT.sh

Slurm

Currently all the above partitions are available under Slurm, however Slurm jobs can only be submitted from Globus. Therefore after logging into Biocluster via ssh, ssh again into globus.

username@pigeon:~$ ssh -XY globus

Submitting Jobs

Non-interactive submission of a SBATCH script:

sbatch SBATCH_SCRIPT.sh

Here is an example of an SBATCH script:

#!/bin/bash -l

#SBATCH --nodes=1
#SBATCH --ntasks=10
#SBATCH --mem-per-cpu=1G
#SBATCH --time=1-00:15:00     # 1 day and 15 minutes
#SBATCH --output=my.stdout
#SBATCH --mail-user=useremail@address.com
#SBATCH --mail-type=ALL
#SBATCH --job-name="just_a_test"
#SBATCH -p intel # This is the default partition, you can use any of the following; intel, batch, highmem, gpu

# Print current date
date

# Load samtools
module load samtools

# Change directory to where you submitted the job from, so that relative paths resolve properly
cd $SLURM_SUBMIT_DIR

# Concatenate BAMs
samtools cat -h header.sam -o out.bam in1.bam in2.bam

# Print name of node
hostname

The above job will request 1 node, 10 task (assumes 1 cpu core per task), 10GB of memory (1GB per task), for 1 day and 15 minutes. All STDOUT will be redirected to a file called "my.stdout" as well as an email sent to the user when the status of the job changes.

Interactive submission:

srun --pty bash -l

If you do not specify a partition then the intel partition is used by default.

Here is a more complete example:

srun --x11 --mem=1gb --cpus-per-task 1 --ntasks 1 --time 10:00:00 --pty bash -l

The above example enables X11 forwarding and requests, 1GB of memory, 1 cores, for 10 hours within an interactive session.

Monitoring Jobs

To check on your jobs states, run the following:

squeue -u username

To list all the details of a specific job, run the following:

scontrol show job <JOBID>

Advanced Jobs

There is a third way of submitting jobs by using steps.

Single Step submission:

srun <command>

Under a single step job your command will hang until appropriate resources are found and when the step command is finished the results will be sent back on STDOUT. This may take some time depending on the job load of the cluster.

Multi Step submission:

salloc -N 4 bash -l
srun <command>
...
srun <command>
exit

Under a multi step job the salloc command will request resources and then your parent shell will be running on the head node. This means that all commands will be executed on the head node unless preceeded by the srun command. You will also need to exit this shell in order to terminate your job.

GPU Jobs

A single GPU job will no longer reserve an entire node. For each node there are 4 GPUs. This means that you need to request how many GPUs that you would like to use.

Non-Interactive:

srun -p gpu --mem=100g --time=1:00:00 SBATCH_SCRIPT.sh


Interactive

srun -p gpu --gres=gpu:4 --mem=100g --time=1:00:00 --pty bash -l


Torque/Maui

Submitting Jobs

The command used to submit jobs is qsub. There are two basic ways to submit jobs:

The first way this can be done is by using a technique where you pipe in the command via STDIN.

For example, lets say you have a set of files in your home directory and want to run blast against them, you could run a command similar to what we find below to have that run on a node.

When using the cluster it quickly becomes useful to be able to run multiple commands as part of a single job. To solve this we write scripts. In this case, the way it works is that we invoke the script as the last argument to qsub.

So if this script was called blast_AE004437.sh we could run the following to make all of those steps happen.

Tracking Jobs

Now that we have a job in the queue, how do I know if it is running? For that, there is a command called qstat. The command qstat will provide you with the current state of all the jobs running or queued to run on the cluster. The following is an example of that output:

Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
467655.torque-server       ...MTcauLSS_5.sh xzhang          2562047: R batch          
467660.torque-server       ...TnormLSS_5.sh xzhang          5124113: R batch          
467663.torque-server       ARSUMTt3LSS_5.sh xzhang          5124113: R batch          
474003.torque-server       Aedes2           bradc           1095:30: R batch          
7475989.torque-server       Culex2           bradc           928:40:0 R batch          
478663.torque-server       STDIN            snohzadeh       00:36:28 R batch          
626327.torque-server       STDIN            wangya          11:16:38 R batch          
645318.torque-server       STDIN            yfu             477:49:3 R batch          
645353.torque-server       STDIN            yfu             464:31:4 R batch          
655060.torque-server       newphyml.sh      nponts          364:57:5 R batch          
655077.torque-server       newphyml.sh      nponts          401:32:2 R batch          
655182.torque-server       newphyml.sh      nponts          396:35:2 R batch          
655385.torque-server       newphyml.sh      nponts          337:29:4 R batch          
655469.torque-server       newphyml.sh      nponts          146:23:5 R batch          
655493.torque-server       newphyml.sh      nponts          335:05:0 R batch          
655571.torque-server       newphyml.sh      nponts          358:33:5 R batch          
655742.torque-server       newphyml.sh      nponts          314:08:5 R batch          
655754.torque-server       newphyml.sh      nponts          299:45:2 R batch          
655814.torque-server       newphyml.sh      nponts          109:59:4 R batch          
655951.torque-server       newphyml.sh      nponts          268:58:3 R batch          
655962.torque-server       newphyml.sh      nponts          325:04:2 R batch          
656054.torque-server       newphyml.sh      nponts          277:43:3 R batch          
656055.torque-server       newphyml.sh      nponts          327:37:2 R batch          
656195.torque-server       newphyml.sh      nponts          270:07:2 R batch          
656309.torque-server       newphyml.sh      nponts          261:18:4 R batch          
656339.torque-server       newphyml.sh      nponts          306:47:0 R batch          
656340.torque-server       newphyml.sh      nponts          275:05:4 R batch          
656486.torque-server       newphyml.sh      nponts          259:59:3 R batch          
659489.torque-server       STDIN            zwu             00:25:01 R batch          
672645.torque-server       STDIN            snohzadeh       00:00:00 R batch          
674351.torque-server       STDIN            yfu             165:40:5 R batch          
674819.torque-server       Seqrank_CL_08.sh xzhang          115:43:3 R batch          
674940.torque-server       submit_script.sh nsausman        683:14:1 R batch          
675260.torque-server       ...eatModeler.sh robb            233:01:5 R batch          
675266[].torque-server     sa.sh            jban                   0 R batch          
675275[].torque-server     LeucoMakerMrctr  hishak                 0 R batch          
675853[].torque-server     sa.sh            jban                   0 R batch          
677089.torque-server       LFPcorun.sh      jychen          57:31:33 R batch          
679437.torque-server       Chr8.mergeBam.sh robb                   0 Q batch          
679438.torque-server       Chr9.mergeBam.sh robb                   0 Q batch          
679439.torque-server       Chr1.cat_fq.sh   robb                   0 Q batch          
679440.torque-server       Chr10.cat_fq.sh  robb                   0 Q batch          
679441.torque-server       Chr11.cat_fq.sh  robb                   0 Q batch          
679442.torque-server       Chr12.cat_fq.sh  robb                   0 Q batch          
... CONTINUED ...

The R in the S column means a job is running and a Q means that the job is queued waiting to run. Jobs get queued for a number of reasons, the most common are:

1. A job scheduling run has not yet completed. Scheduling runs take place approximately every 15 seconds.
2. The queue is at ~75% capacity and the job is requesting a significant amount of walltime.
3. The queue is at 100% capacity and the job has no place it can be started.
4. The job is requesting specific resources, such as 8 processors, and there is no place the system is able to fit it.
5. The user submitting the job has reached a resource maximum for that queue and cannot start any more jobs running until other jobs have finished.

There are additional flags that can be passed to qstat to get more information about the state of the cluster, including the -u flag that will only display the status of jobs for a particular user.
Once a job has finished, it will no longer show up in this listing. 

Job Results

By default, results from the jobs come out two different ways.

1. The system sends STDOUT and STDERR to files called <job_name>.o<job_number> and <job_name>.e<job_number>.
2. Any output created by your script, like the blastp.out in the example above.

For example if you ran the example from above and got a job number of 679746, you would end up with a file called blast_AE004437.sh.o679746 and a file called blast_AE004437.sh.e679746 in the directory where you ran qsub. Additionally, because our script creates a directory using the PBS_JOBID variable, you would have a directory in your home directory called 679746.torque01.

Deleting Jobs

Sometimes you need to delete a job. You may need to do this if you accidentally submitted something that will run longer than you want or perhaps you accidentally submitted the wrong script. To do delete a job, you use the qdel command. If you wanted to delete job number 679440, you would run:

Please be aware that you can only delete jobs that you own.

Delete all jobs of one user:

Delete all jobs running by one user:

Delete all jobs queued jobs by one user:

Advanced Usage

There are number of additional things you can do with qsub that do a better job of taking advantage of the cluster.

man qstat 

Requesting Additional Resources

Frequently, there is a need to use more than one processor or to specify some amount of memory. The qsub command has a -l flag that allows you to do just that.

Example: Requesting A Single Node with 8 Processors

Let's assume that the script we used above was multi-threaded and spins up 8 different processes to do work. If you wanted to ask for the processors required to do that, you would run the following:

This tells the system that your job needs 8 processors and it allocates them to you.

Example: Requesting 16GB of RAM for a Job

Using the same script as above, let's instead assume that this is just a monolitic process but we know that it will need about 16GB of RAM. Below is an example of how that is done:

Example: Requesting 2 Weeks of Walltime for a Job
Using the same script as above, let's instead assume that it is going to run for close to 2 weeks. We know there are 7 days in a week and 24 hours in a day, so 2 weeks in hours would be (2 * 7 *24) 336 hours. Below is an example of requesting that a job can run for 336 hours.

Example: Requesting Specific Node(s)
The following requests 8 CPU cores and 16GB of RAM on a high memory node for 220 hours:

Interactive Jobs

Sometimes, when testing, it is useful to run commands interactively instead of with a script. To do this you would run:

Just like scripts though, you may need additional resources. To solve this, specify resources, just like you would above:

GPU Jobs

Testing the Queueing System

The following submits 4 jobs to the queue each running 20 seconds. Note: the '-o' and '-e' arguments are assigned '/dev/null', meaning the standard output/error stream files will be omitted. 

for i in {1..4}; do echo "sleep 20" | qsub -q batch -l nodes=1:ppn=1,mem=4gb,walltime=02:00:00 -o /dev/null -e /dev/null; done;

Array Jobs

Many tasks in Bioinformatics need to be parallelized to be efficient. One of the ways we address this is using array jobs. An array job executes the same script a number of times depending on what arguments are passed. To specify that an array should be used, you use the -t flag. For example, if you wanted a ten element array, you would pass -t 1-10 to qsub. You can also specify arbitrary numbers in the array. Assume for a second that the 3 and 5-7 jobs failed for some unknown reason in your last run, you can specify -t 3,5-7 and run just those array elements.

Below is an example that does the same thing that the basic example from above, except that it spreads the workload out into seven different processes. This technique is particularly useful when dealing with much larger datasets.

Prepare Dataset

The script below creates a working directory and builds out the usable dataset. The job is passed to qsub with no arguments.

#!/bin/bash

# Create a directory for us to do work in.
# We are using a special variable that is set by the cluster when a job runs.

mkdir blast_AE004437

# Change to that new directory
cd blast_AE004437

# Copy the proteome of Halobacterium spec.
cp /srv/projects/db/ex/AE004437.faa .

# Do some basic analysis

# The echo command prints info to our output file
echo "How many predicted proteins are there?"
grep '^>' AE004437.faa --count

echo "How many proteins contain the pattern \"WxHxxH\" or \"WxHxxHH\"?"
egrep 'W.H..H{1,2}' AE004437.faa

# Start preparing to do a blast run

# Use awk to grab a number of proteins and then put them in a file.
echo "Generating a set of IDs"
awk --posix -v RS='>' '/W.H..(H){1,2}/ { print ">" $0;}' AE004437.faa | grep '^>' | awk --posix -v FS='|' '{print $4;}' > my_IDs

# Make the proteome blastable
echo "Making a blastable database"
formatdb -i AE004437.faa -p T -o

Analyze the Dataset

The script below will do the actual analysis. Assuming the name is blast_AE004437-multi.sh, the command to submit it would be qsub -t 1-7 blast_AE004437-multi.sh.

#!/bin/bash

# Specify the number of array runs. This means we are going to specify -t 1-7
# when calling qsub.
NUM=7

# Change to that new directory
cd blast_AE004437

# Do some math based on the number of runs we are going to do to figure out how
# many lines, and which lines should be in this run.
LINES=`cat my_IDs | wc -l`
MULTIPLIER=$(( $LINES / $NUM ))
SUB=$(( $MULTIPLIER - 1 ))

END=$(( $PBS_ARRAYID * $MULTIPLIER ))
START=$(( $END - $SUB ))

# Grab the IDs that are going to be part of each blast run
awk "NR==$START,NR==$END" my_IDs > $PBS_ARRAYID.IDs

# Make blastable IDs
echo "Making a set of blastable IDs"
fastacmd -d AE004437.faa -i $PBS_ARRAYID.IDs > $PBS_ARRAYID.fasta

# Run blast
echo "Running blast"
blastall -p blastp -i $PBS_ARRAYID.fasta -d AE004437.faa -o $PBS_ARRAYID.blastp.out -e 1e-6 -v 10 -b 10

Specifying Queues

Troubleshooting

showbf -f batch





showstart JOBID





showhold | grep JOBID





To quickly check your usage and quota limits: 

check_quota home
check_quota bigdata

To get the usage of your current directory, run the following command:

To calculate the sizes of each separate sub directory, run: 

du -shc *

This may take some time to complete, please be patient.

For more information on your home directory, please see the Orientation section in the  Linux Basics manual.

Table of Contents

If you wanted to share this file with someone outside the 'operations' group, read permissions must be added to the file for 'other'.

Set Default Permissions

In Linux, it is possible to set the default file permission for new files. This is useful if you are collaborating on a project, or frequently share files and  you do not want to be constantly adjusting permissions  The command responsible for this is called 'umask'. You should first check what your default permissions currently are by running 'umask -S'.

To set your default permissions, simply run umask with the correct options. Please note, that this does not change permissions on any existing files, only new files created after you update the default permissions. For instance, if you wanted to set your default permissions to you having full control, your group being able to read and execute your files, and no one else to have access, you would run:

It is also important to note that these settings only affect your current session. If you log out and log back in, these settings will be reset. To make your changes permanent you need to add them to your '.bashrc' file, which is a hidden file in your home directory (if you do not have a '.bashrc' file, you will need to create an empty file called '.bashrc' in your home directory). Adding umask to your .bashrc file is as simple as adding your umask command (such as 'umask u=rwx,g=rx,o=r') to the end of the file. Then simply log out and back in for the changes to take affect. You can double check that the settings have taken affect by running 'umask -S'.

Further Reading

Table of Contents

Communicating with Other Users

Looking-Up Specific Users

Listing Users with Active Jobs on the Cluster

R libraries