IIGB Linux Cluster

Data Storage

Memory

Login

The initial log-in, brings users into the biocluster head node. From there, users can submit jobs via qsub to the compute nodes or log into owl to perform memory intensive tasks. Since all machines are mounting a centralized file system, users will always see the same home directory on all systems. Therefore, there is no need to copy files from one machine to another.

Login from Mac or Linux

Open the terminal and type:

Login from Windows

What's Next?

After the login users have the following options for submitting compute tasks:

• Submit computationally intensive tasks via qsub to the cluster nodes.
• Run memory intensive jobs on owl. The command 'ssh owl' logs users into this dedicated high-memory node.
  
• Please do not run ANY computationally intensive tasks on the head node. If this is done, we will have to kill your jobs, because they will slow down all other users.

Modules

The modules system is a way to easily load software into your path. This approach has a number of advangates including allowing for multiple versions of the software to be installed at any given time.

Listing Available Software

To list the available software run:

module avail

This should output something like:

------------------------- /usr/local/Modules/versions --------------------------
3.2.9

--------------------- /usr/local/Modules/3.2.9/modulefiles ---------------------
BEDTools/2.15.0(default)     modules
PeakSeq/1.1(default)         python/3.2.2
SOAP2/2.21(default)          samtools/0.1.18(default)
bowtie2/2.0.0-beta5(default) stajichlab
cufflinks/1.3.0(default)     subread/1.1.3(default)
matrix2png/1.2.1(default)    tophat/1.4.1(default)
maui/3.3.1(default)          trans-ABySS/1.2.0(default)
module-info

Using The Software

To load a module into your path, run:

module load <software name>[/<version>]

You only need to add the version if you are wanting to select a version other than default. So, if I wanted to load tophat, I would run:

module load tophat

If I wanted to specifically load version 1.4.1, I would run:

module load tophat/1.4.1

Showing What Software is Loaded

To show what modules you have loaded at any time, you can run:

module list

Depending on what modules you have loaded, it will produce something like this:

Currently Loaded Modulefiles:
  1) maui/3.3.1     2) tophat/1.4.1   3) PeakSeq/1.1

Unloading Software

Sometimes you want to no longer have a piece of software in path. To do this you unload the module by running:

module unload <software name>

Additional Features

There are additional features and operations that can be done with the module command. Please run the following to get more information:

module help

Managing Jobs

Submitting and managing jobs is a the heart of using the cluster. Jobs are the way that you run software on the nodes in the cluster.

Usage Guidelines

There are a number of different queues available to cluster users. Below is a table of the resource limitations associated with each:

• batch - This is the default queue, users may occupy no more than 64 processors at a time. This uses nodes n01-n34. A default walltime of 168 hours or 7 days is set on all jobs started in this queue.To use more than this, please use the -l flag in qsub.
• loprio - This queue is for use with lots of short lived jobs. It has a wall clock limit of 10 minutes and the priority is set so that jobs in the batch queue will run before jobs in this queue. There is no limit on the processors that can be used. This queue uses nodes n01-n34.
• highmem - This queue is for jobs that require 16GB or more of RAM. By default, all jobs will be set to a limit of 16GB of RAM and a wall clock limit of 6 hours. To use more than this, please use the -l flag in qsub.

Basic Usage

Submitting Jobs

The command used to submit jobs is qsub. There are two basic ways to submit jobs:

Using STDIN

The first way this can be done is by using a technique where you pipe in the command via STDIN.

echo <command> | qsub

For example, lets say you have a set of files in your home directory and want to run blast against them, you could run a command similar to what we find below to have that run on a node.

echo blastall -p blastp -i myseq.fasta -d AE004437.faa -o blastp.out -e 1e-6 -v 10 -b 10 | qsub

Using a Script

When using the cluster it quickly becomes useful to be able to run multiple commands as part of a single job. To solve this we write scripts. In this case, the way it works is that we invoke the script as the last argument to qsub.

qsub <script name>

A script is just a set of commands that we want to make happen once the job runs. Below is an example script that does the same thing that we do with Exercise 5 in the Linux BasicsManual.

#!/bin/bash

# Create a directory for us to do work in.
# We are using a special variable that is set by the cluster when a job runs.
mkdir $PBS_JOBID

# Change to that new directory
cd $PBS_JOBID

# Copy the proteome of Halobacterium spec.
cp /srv/projects/db/ex/AE004437.faa .

# Do some basic analysis

# The echo command prints info to our output file
echo "How many predicted proteins are there?"
grep '^>' AE004437.faa --count

echo "How many proteins contain the pattern \"WxHxxH\" or \"WxHxxHH\"?"
egrep 'W.H..H{1,2}' AE004437.faa

# Start preparing to do a blast run

# Use awk to grab a number of proteins and then put them in a file.
echo "Generating a set of IDs"
awk --posix -v RS='>' '/W.H..(H){1,2}/ { print ">" $0;}' AE004437.faa | grep '^>' | awk --posix -v FS='|' '{print $4;}' > my_IDs

# Make the proeome blastable
echo "Making a blastable database"
formatdb -i AE004437.faa -p T -o

# Make blastable IDs
echo "Making a set of blastable IDs"
fastacmd -d AE004437.faa -i my_IDs > myseq.fasta

# Run blast
echo "Running blast"
blastall -p blastp -i myseq.fasta -d AE004437.faa -o blastp.out -e 1e-6 -v 10 -b 10

So if this script was called blast_AE004437.sh we could run the following to make all of those steps happen.

qsub blast_AE004437.sh

Tracking Jobs

Now that we have a job in the queue, how do I know if it is running? For that, there is a command called qstat. The command qstat will provide you with the current state of all the jobs running or queued to run on the cluster. The following is an example of that output:

Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
467655.torque-server       ...MTcauLSS_5.sh xzhang          2562047: R batch          
467660.torque-server       ...TnormLSS_5.sh xzhang          5124113: R batch          
467663.torque-server       ARSUMTt3LSS_5.sh xzhang          5124113: R batch          
474003.torque-server       Aedes2           bradc           1095:30: R batch          
7475989.torque-server       Culex2           bradc           928:40:0 R batch          
478663.torque-server       STDIN            snohzadeh       00:36:28 R batch          
626327.torque-server       STDIN            wangya          11:16:38 R batch          
645318.torque-server       STDIN            yfu             477:49:3 R batch          
645353.torque-server       STDIN            yfu             464:31:4 R batch          
655060.torque-server       newphyml.sh      nponts          364:57:5 R batch          
655077.torque-server       newphyml.sh      nponts          401:32:2 R batch          
655182.torque-server       newphyml.sh      nponts          396:35:2 R batch          
655385.torque-server       newphyml.sh      nponts          337:29:4 R batch          
655469.torque-server       newphyml.sh      nponts          146:23:5 R batch          
655493.torque-server       newphyml.sh      nponts          335:05:0 R batch          
655571.torque-server       newphyml.sh      nponts          358:33:5 R batch          
655742.torque-server       newphyml.sh      nponts          314:08:5 R batch          
655754.torque-server       newphyml.sh      nponts          299:45:2 R batch          
655814.torque-server       newphyml.sh      nponts          109:59:4 R batch          
655951.torque-server       newphyml.sh      nponts          268:58:3 R batch          
655962.torque-server       newphyml.sh      nponts          325:04:2 R batch          
656054.torque-server       newphyml.sh      nponts          277:43:3 R batch          
656055.torque-server       newphyml.sh      nponts          327:37:2 R batch          
656195.torque-server       newphyml.sh      nponts          270:07:2 R batch          
656309.torque-server       newphyml.sh      nponts          261:18:4 R batch          
656339.torque-server       newphyml.sh      nponts          306:47:0 R batch          
656340.torque-server       newphyml.sh      nponts          275:05:4 R batch          
656486.torque-server       newphyml.sh      nponts          259:59:3 R batch          
659489.torque-server       STDIN            zwu             00:25:01 R batch          
672645.torque-server       STDIN            snohzadeh       00:00:00 R batch          
674351.torque-server       STDIN            yfu             165:40:5 R batch          
674819.torque-server       Seqrank_CL_08.sh xzhang          115:43:3 R batch          
674940.torque-server       submit_script.sh nsausman        683:14:1 R batch          
675260.torque-server       ...eatModeler.sh robb            233:01:5 R batch          
675266[].torque-server     sa.sh            jban                   0 R batch          
675275[].torque-server     LeucoMakerMrctr  hishak                 0 R batch          
675853[].torque-server     sa.sh            jban                   0 R batch          
677089.torque-server       LFPcorun.sh      jychen          57:31:33 R batch          
679437.torque-server       Chr8.mergeBam.sh robb                   0 Q batch          
679438.torque-server       Chr9.mergeBam.sh robb                   0 Q batch          
679439.torque-server       Chr1.cat_fq.sh   robb                   0 Q batch          
679440.torque-server       Chr10.cat_fq.sh  robb                   0 Q batch          
679441.torque-server       Chr11.cat_fq.sh  robb                   0 Q batch          
679442.torque-server       Chr12.cat_fq.sh  robb                   0 Q batch          
... CONTINUED ...

The R in the S column means a job is running and a Q means that the job is queued waiting to run. Jobs get queued for a number of reasons, the most common are:

1. A scheduling run has not yet occured to start the job. Scheduling runs take place approximately every 15 seconds.
2. The queue is at 100% capacity and the job has no place it can be started.
3. The user submitting the job has reached a resource maximum for that queue and cannot start any more jobs running until other jobs have finished.
4. The job is requesting specific resources, such as 8 processors, and there is no place the system is able to fit it.

Once a job has finished, it will no longer show up in this listing. There are additional flags that can be passed to qstat to get more information about the state of the cluster, including the -u flag that will only display the status of jobs for a particular user. Please see the qstat manual by running man qstat or by visiting http://www.clusterresources.com/torquedocs21/commands/qstat.shtml.

Job Results

By default, results from the jobs come out two different ways.

1. The system sends STDOUT and STDERR to files called <job_name>.o<job_number> and <job_name>.e<job_number>.
2. Any output created by your script, like the blastp.out in the example above.

For example if you ran the example from above and got a job number of 679746, you would end up with a file called blast_AE004437.sh.o679746 and a file called blast_AE004437.sh.e679746 in the directory where you ran qsub. Additionally, because our script creates a directory using the PBS_JOBID variable, you would have a directory in your home directory called 679746.torque_server.

Deleting Jobs

Sometimes you need to delete a job. You may need to do this if you accidentally submitted something that will run longer than you want or perhaps you accidentally submitted the wrong script. To do delete a job, you use the qdel command. If you wanted to delete job number 679440, you would run:

qdel 679440

Please be aware that you can only delete jobs that you own.

Delete all jobs of one user:

qselect -u $USER | xargs qdel

Delete all jobs running by one user:

qselect -u $USER -s R | xargs qdel

Delete all jobs queued jobs by one user:

qselect -u $USER -s Q | xargs qdel

Advanced Usage

There are number of additional things you can do with qsub that do a better job of taking advantage of the cluster. Below are a few specific examples but it is worth looking at the manual by running man qsub or by visiting http://www.clusterresources.com/torquedocs21/commands/qsub.shtml.

Requesting Additional Resources

Frequently, there is a need to use more than one processor or to specify some amount of memory. The qsub command has a -l flag that allows you to do just that.

Example: Requesting A Single Node with 8 Processors

Let's assume that the script we used above was multi-threaded and spins up 8 different processes to do work. If you wanted to ask for the processors required to do that, you would run the following:

qsub -l nodes=1:ppn=8 blast_AE004437.sh

This tells the system that your job needs 8 processors and it allocates them to you.

Example: Requesting 16GB of RAM for a Job

Using the same script as above, let's instead assume that this is just a monolitic process but we know that it will need about 16GB of RAM. Below is an example of how that is done:

qsub -l mem=16gb blast_AE004437.sh 

Example: Requesting 2 Weeks of Walltime for a Job
Using the same script as above, let's instead assume that it is going to run for close to 2 weeks. We know there are 7 days in a week and 24 hours in a day, so 2 weeks in hours would be (2 * 7 *24) 336 hours. Below is an example of requesting that a job can run for 336 hours.
qsub -l walltime=336:00:00 blast_AE004437.sh
Example: Requesting Specific Node(s)
The following requests 8 CPU cores and 16GB of RAM on high memory node m01 for 220 hours:
qsub -q highmem -l nodes=1:ppn=8+m01,mem=16gb,walltime=220:00:00 assembly.sh

Interactive Jobs

Sometimes, when testing, it is useful to run commands interactively instead of with a script. To do this you would run:

qsub -I

Just like scripts though, you may need additional resources. To solve this, specify resources, just like you would above:

qsub -l mem=16gb -I

Array Jobs

Many tasks in Bioinformatics need to be parallelized to be efficient. One of the ways we address this is using array jobs. An array job executes the same script a number of times depending on what arguments are passed. To specify that an array should be used, you use the -t flag. For example, if you wanted a ten element array, you would pass -t 1-10 to qsub. You can also specify arbitrary numbers in the array. Assume for a second that the 3 and 5-7 jobs failed for some unknown reason in your last run, you can specify -t 3,5-7 and run just those array elements.

Below is an example that does the same thing that the basic example from above, except that it spreads the workload out into seven different processes. This technique is particularly useful when dealing with much larger datasets.

Prepare Dataset

The script below creates a working directory and builds out the usable dataset. The job is passed to qsub with no arguments.

#!/bin/bash

# Create a directory for us to do work in.
# We are using a special variable that is set by the cluster when a job runs.

mkdir blast_AE004437

# Change to that new directory
cd blast_AE004437

# Copy the proteome of Halobacterium spec.
cp /srv/projects/db/ex/AE004437.faa .

# Do some basic analysis

# The echo command prints info to our output file
echo "How many predicted proteins are there?"
grep '^>' AE004437.faa --count

echo "How many proteins contain the pattern \"WxHxxH\" or \"WxHxxHH\"?"
egrep 'W.H..H{1,2}' AE004437.faa

# Start preparing to do a blast run

# Use awk to grab a number of proteins and then put them in a file.
echo "Generating a set of IDs"
awk --posix -v RS='>' '/W.H..(H){1,2}/ { print ">" $0;}' AE004437.faa | grep '^>' | awk --posix -v FS='|' '{print $4;}' > my_IDs

# Make the proteome blastable
echo "Making a blastable database"
formatdb -i AE004437.faa -p T -o

Analyze the Dataset

The script below will do the actual analysis. Assuming the name is blast_AE004437-multi.sh, the command to submit it would be qsub -t 1-7 blast_AE004437-multi.sh.

#!/bin/bash

# Specify the number of array runs. This means we are going to specify -t 1-7
# when calling qsub.
NUM=7

# Change to that new directory
cd blast_AE004437

# Do some math based on the number of runs we are going to do to figure out how
# many lines, and which lines should be in this run.
LINES=`cat my_IDs | wc -l`
MULTIPLIER=$(( $LINES / $NUM ))
SUB=$(( $MULTIPLIER - 1 ))

END=$(( $PBS_ARRAYID * $MULTIPLIER ))
START=$(( $END - $SUB ))

# Grab the IDs that are going to be part of each blast run
awk "NR==$START,NR==$END" my_IDs > $PBS_ARRAYID.IDs

# Make blastable IDs
echo "Making a set of blastable IDs"
fastacmd -d AE004437.faa -i $PBS_ARRAYID.IDs > $PBS_ARRAYID.fasta

# Run blast
echo "Running blast"
blastall -p blastp -i $PBS_ARRAYID.fasta -d AE004437.faa -o $PBS_ARRAYID.blastp.out -e 1e-6 -v 10 -b 10

Using the highmem and lowprio Queues

These queues provide access to additional resources or allow use of resources in different ways. To take advantage of the queues, you will need to do specify the -q flag with the queue name on the command line.

qsub -q highmem

or

qsub -q lowprio

Data Storage

Storage Locations

Home Directories

Home directories are where you place the scripts and various things you are working on, on biocluster. This space is very limited. Please see the Quotas section above for the space that is allocated per user.

Big Data

Big data is an area where large amounts of storage can be made available to users. A lab purchases big data space seperately from access to the cluster. This space is then made available to the lab via a shared directory and individual directories for each user.

Lab Shared Space

This directory can be accessed by the lab as a whole.

Individual User Space

This directory can be accessed by specific lab members.

Non-Persistent Space

Frequently, there is a need to do things like, output a signifigant amount of intermediate data durring a job, access a dataset from a faster medium than bigdata or the home directories or write out lock files. These types of things are well suited to the use of non-persistent spaces. Below are the filesystems available on biocluster.

Memory Backed Space

This type of space takes away from physical memory but allows extremely fast access to the files located on it. You will need to factor in the space you are using in RAM as well. For example, if you have a dataset that is 1G in size and use this space, it will take 1G of RAM.

Temporary Space

This is the standard space available on all Linux systems. Please be aware that it is limited to the amount of free disk space on the node you are running on.

SSD Backed Space

This space is must faster than the standard temporary space, but slower than being memory backed.

Sharing data with other users

Copying large folders to and from Biocluster

• Copy (transfer) folders between different storage hardware
• Perform transfers over the network via SSH
  
• Compare large data sets (-n, --dry-run option)
• Resume interrupted transfers
  

• Rsync does not exist on Windows. Only Mac and Linux support rsync natively.
• Always put the / after both folder names, e.g: FOLDER_B/ Failing to do so will result in the nesting folders every time you try to resume. If you don't put / you will get a second folder_B inside folder_B  FOLDER_B/FOLDER_B/
• Rsync does not move but only copies.
• man rsync
  

Copying large folders on Biocluster between Directories

• Once the rsync command is done, run it again. The second run will be short and it just a check. If there was no output, nothing changed, it is safe to delete the original location.
  Specifically, running rsync the second time will ensure that everything has been transferred correctly. The -i (--itemize-changes) option asks rsync to report (output) all the changes that occure on to the filesystem during the sync. No output = No changes = The folder has been transfered safely.
  
• All the bullets in the above section (Copying large folders to and from Biocluster) apply to this section
  

Copying large folders between Biocluster and other servers

• All the bullets in the above sections (Copying large folders to and from Biocluster) apply to this section

Table of Contents

Communicating with Other Users

Looking-Up Specific Users

Listing Users with Active Jobs on the Cluster

R libraries 

Password Change

Changing your password is currently disabled for security reasons

You can keep using the original Biocluster random password or change it:

1. Log-in via SSH using the Terminal on Mac/Linux or Putty on Windows
2. Type passwd
3. Enter the old password (the random characters that you were given as your initial passowrd)
4. Enter your new password

The password minimum length is:

• 10 lower-case letters
  
• 8 letters AND 1 digit (9 characters total)
• 8 letters AND 1 upper-case (9 characters total)
• 8 letters AND 1 non-alphanumeric¹ (9 characters total)
• 6 letters AND 1 digit AND 1 upper-case (8 characters total)
• 6 letters AND 1 digit AND 1 non-alphanumeric¹ (8 characters total)
• 6 letters AND 1 upper-case AND 1 non-alphanumeric¹ (8 characters total)
• 4 letters AND 1 digit AND 1 upper-case AND 1 non-alphanumeric¹ (7 characters total)