BX:SGE

Overview

There is one central SGE installation which handles job scheduling across all of the BX clusters, work servers, and workstations (with the exception of the okinawa, linne, and galaxy clusters). Merging existing clusters, work servers, and workstations is still a work-in-progress project.

The central grid engine has a pair of fully redundant master servers to ensure continuous job scheduling. The loss of both sge masters does not kill jobs that are currently running or queued, but will prevent any further job submissions. There is an approximately 5 minute failover period between sge master failure and the startup of the other sge master.

SGE can be used from any BX-maintained machine. Use `which qsub` to determine if you have access to the BX SGE grid. If not, try sourcing /afs/bx.psu.edu/service/sge/prod/default/common/settings.sh if you're using bash, or /afs/bx.psu.edu/service/sge/prod/default/common/settings.csh if you're using tcsh. If you see a message saying your machine is not an administrative host or submission host, send an email to admin-at-bx.psu.edu with the name of the host to have it added.

SGE is less of a queue system, and more of a resource broker. You tell it what your jobs need to run, and it finds the best machines to run them on.

Status

• Current BX Grid load can be seen through GANGLIA at http://ganglia.bx.psu.edu
• A web version of qstat (XSL formatted version of qstat -f -u '*' -xml) is available at http://qstat.bx.psu.edu

Usage

Job submission

To submit a job, put the command(s) into a script, and use qsub.

Various job resource requirements can be specified with -l resource=foo. Careful consideration should be given to your job's resource requirements. Specifying arch, mem_free, s_vmem, and slots for threaded jobs is essential to ensure your job does not over-subscribe on resources, and runs to completion. SGE cannot predict what resources your job requires, for example it cannot predict how much memory your job will require, so it might schedule it on a node that has far less memory than necessary, causing the node to wedge itself or the job to die.

For more detailed usage and examples, please see the SGE Documentation Site: SGE 6.2u5 documentation

Job monitoring

SGE host status can be seen with qhost

Job queue/status can be seen with qstat -f, which will show just your jobs. To see everyone's jobs, qstat -f -u '*'. Note that qstat behaves different than previous versions of SGE, where the new default is to show only your jobs.

It is highly recommended that job notification be turned on depending on how many jobs you will be submitting. This can be done with the SGE variable settings as seen below:

#$ -M youremail@bx.psu.edu
#$ -m beas

This will send email when your job starts, finishes, is suspended, or aborted or rescheduled.

Queue permissions

Anyone can submit jobs to all.q. However, permission is restricted at the host and hostgroup level. There are user groups that control who's jobs can run on particular groups of hosts. There are also groups to further restrict access to individual hosts within a hostgroup.

Currently, the following host groups are defined, along with the user group that can access them:

• Persephone cluster (c1-14) : persephone_users
• Zhang Lab (townsend) : zhang_lab
• c14 : c14_users

Complexes

See complex(5) and sge_types(1) for full explanations.

At job submission time, you should specify the job requirements. The better you can describe the requirements, the better Grid Engine will be able to find a host or series of hosts to run your job. If you don't do this, your job will more than likely run out of memory, not have enough CPU, not have enough scratch space, run on a node that has a slow network connection, etc.

Complexes can be specified in a combination of command line arguments and submission script comments. In a qsub script:

#$ -l arch=lx24-amd64
#$ -l mem_tokens=1G
#$ -l mem_free=1G

some_command

As command arguments:

qsub -l arch=lx24-amd64 -l mem_tokens=1G -l mem_free=1G myscript.qsub

Some of the more common complexes that are used:

complex	type	consumable	description	example
arch	regex string	no	Specifies CPU architecture and OS	lx24-amd64
mem_tokens	memory	yes	Think of this as a queue slot, but for memory. Should be used along with mem_free for best results.	2G
mem_free	memory	no	Specifies amount of free memory on a node. Should be used along with mem_tokens for best results.	2G
netspeed	memory	no	Specifies speed of primary network interface	1g, 100m
scratch_free	memory	no	Amount of free space in /scratch. Updated every 5 minutes. Only observed at job start time	50G

Memory values:

• lowercase suffixes denote powers of 1000 (k, m, g).
• uppercase suffixes denote powers of 1024 (K, M, G).

Use qhost -F [-h <hostname>] to see current complex values for a particular host. Example output:

$ qhost -F -h townsend
HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
-------------------------------------------------------------------------------
global                  -               -     -       -       -       -       -
townsend.bx.psu.edu     lx24-amd64      2  0.00    3.7G    2.4G    1.0G    1.0G
   hl:arch=lx24-amd64
   hl:num_proc=2.000000
   hl:mem_total=3.739G
   hl:swap_total=1.004G
   hl:virtual_total=4.742G
   hl:load_avg=0.000000
   hl:load_short=0.000000
   hl:load_medium=0.000000
   hl:load_long=0.000000
   hl:mem_free=1.366G
   hl:swap_free=3.199M
   hl:virtual_free=1.369G
   hl:mem_used=2.372G
   hl:swap_used=1.000G
   hl:virtual_used=3.373G
   hl:cpu=0.100000
   hl:m_topology=SCC
   hl:m_topology_inuse=SCC
   hl:m_socket=1.000000
   hl:m_core=2.000000
   hl:np_load_avg=0.000000
   hl:np_load_short=0.000000
   hl:np_load_medium=0.000000
   hl:np_load_long=0.000000
   hl:netspeed=95.367M
   hl:scratch_free=884.541G
   hl:scratch_total=884.548G
   hl:scratch_used=7.233M
   hc:mem_tokens=4.000G

Note: complex reporting is always done in powers of 1024. In this case, netspeed is 95.367M, or 95.367*2^(2*10). To get this back to decimal bits from binary bits, we simply multipley 95.367 by 1024*1024, then divide by 1000*1000 to get 99.99m, or 100m, so we see that this machine is on a 100mbit connection.

Disk space

Most if not all nodes have a /scratch directory which should be used in lieu of /tmp or /var/tmp for temporary job output.

You can specify -l scratch_free=xG where x is the amount of free space in /scratch.

AFS considerations

Without going into too much technical detail, when you submit a job, SGE 'steals' your Kerberos tickets and sends them along with the job, and uses them to obtain AFS tokens right before the job starts. Your Kerberos tickets, and hence your AFS tokens, have a default lifetime of 14 days, renewable up to 30 days. It is essential that you pay attention to your current ticket expiration date before submitting a long running job, or a job that will sit in the queue for a significant period of time. Ticket expiration time can be seen in the output of klist.

If you submit a job without currently posessing valid Kerberos tickets for your user, then your job will not have authenticated access to AFS, and hence your home directory.

You should not use your AFS home directory for any high throughput IO. The AFS home directory servers are fast, but are certainly not fast enough to serve HPC needs (with the exception of the dedicated bare-hardware AFS bulk storage servers). Running programs, scripts, reading small files, writing small output files should be fine, though. AFS does tend to break down when there are multiple writers to the same volume/directory, so keep this in mind. In most cases, the various NFS storage systems should be preferred to AFS.