Difference between revisions of "SLab:File System Layout"

From CCGB
Jump to: navigation, search
 
Line 6: Line 6:
 
The volume mounted at this location is RO-replicated. The main purpose of this root data.schuster_lab volume is to contain mounpoints to other volumes, as well as the symlink farm under ''sequencing/''
 
The volume mounted at this location is RO-replicated. The main purpose of this root data.schuster_lab volume is to contain mounpoints to other volumes, as well as the symlink farm under ''sequencing/''
  
= Sequencing Directory =
+
== scratch/ ==
 +
Individual global scratch volumes. On request only (admin-at-bx.psu.edu).
 +
 
 +
== sequencing/ ==
  
 
* /afs/bx.psu.edu/depot/data/schuster_lab/sequencing
 
* /afs/bx.psu.edu/depot/data/schuster_lab/sequencing
Line 15: Line 18:
  
 
SSDFS is not currently capable of delegating commands to normal users, and requires some finesse when executing certain commands due to NFS permissions. Email admin-at-bx.psu.edu if something under sequencing/ doesn't look right.
 
SSDFS is not currently capable of delegating commands to normal users, and requires some finesse when executing certain commands due to NFS permissions. Email admin-at-bx.psu.edu if something under sequencing/ doesn't look right.
 +
 +
In short, datasets are organized into "volumes" distributed between 2 servers and multiple NFS filesystems. 454 datasets are organized by one volume per month, and Illumina datasets are one volume per dataset. SSDFS fully abstracts the actual location of the volume (server/filesystem). If you need direct access to a volume, it will be under ''/afs/bx.psu.edu/depot/data/schuster_lab/.ssdfs/vol/by-name/<name>''
  
 
Layout:
 
Layout:
 
* /afs/bx.psu.edu/depot/data/schuster_lab/sequencing/
 
* /afs/bx.psu.edu/depot/data/schuster_lab/sequencing/
** 454/
+
** 454/ (all 454 runs)
 
*** $YEAR/
 
*** $YEAR/
 
**** $YEAR/$YEAR-$MONTH/
 
**** $YEAR/$YEAR-$MONTH/
 
*** incoming/
 
*** incoming/
** illumina/
+
** illumina/ (all illumina runs)
 
*** $YEAR/
 
*** $YEAR/
 
**** $YEAR/$RUN_NAME/
 
**** $YEAR/$RUN_NAME/
Line 31: Line 36:
 
**** illumina-5 -> /nfs/s2.persephone.bx.psu.edu/md1k-5-data/illumina/
 
**** illumina-5 -> /nfs/s2.persephone.bx.psu.edu/md1k-5-data/illumina/
 
**** illumina-6 -> /nfs/s2.persephone.bx.psu.edu/md1k-6-data/illumina/
 
**** illumina-6 -> /nfs/s2.persephone.bx.psu.edu/md1k-6-data/illumina/
 +
** scripts/ (Various shell scripts, qsub scripts, etc)
 +
*** 454/
 +
** support/
 +
*** 454/
 +
*** illumina/
  
== support ==
+
== support/ ==
 
* /afs/bx.psu.edu/depot/data/schuster_lab/sequencing/support/
 
* /afs/bx.psu.edu/depot/data/schuster_lab/sequencing/support/
** 454/
 
** illumina/
 
  
 
The sequencing support directory contains manuals, software, and reference genomes used during sequencing and sequence processing.
 
The sequencing support directory contains manuals, software, and reference genomes used during sequencing and sequence processing.
  
= Projects Directory =  
+
= projects/ =  
 
* /afs/bx.psu.edu/depot/data/schuster_lab/projects
 
* /afs/bx.psu.edu/depot/data/schuster_lab/projects
  
 
The projects directory contains directories for project specific data including the Tasmanian Devil, KB1, and Woolly Mammoth projects.
 
The projects directory contains directories for project specific data including the Tasmanian Devil, KB1, and Woolly Mammoth projects.
  
= Users Directory =
+
= users/ =
 
* /afs/bx.psu.edu/depot/data/schuster_lab/users
 
* /afs/bx.psu.edu/depot/data/schuster_lab/users
  
 
The users directory contains directories for people within the Schuster Lab.
 
The users directory contains directories for people within the Schuster Lab.

Latest revision as of 16:47, 4 October 2010

Schuster Lab Root Directory

  • /afs/bx.psu.edu/depot/data/schuster_lab

The Schuster Lab's data can be found in the /afs/bx.psu.edu/depot/data/schuster_lab directory.

The volume mounted at this location is RO-replicated. The main purpose of this root data.schuster_lab volume is to contain mounpoints to other volumes, as well as the symlink farm under sequencing/

scratch/

Individual global scratch volumes. On request only (admin-at-bx.psu.edu).

sequencing/

  • /afs/bx.psu.edu/depot/data/schuster_lab/sequencing

Access to all of the Schuster Lab's sequencing data is through the sequencing directory. This directory contains archived run directories as well as directories for runs that are currently in progress.

Most of the top-level run directories under sequencing/ are symlinks managed by the new symlink management system: http://github.com/phalenor/ssdfs

SSDFS is not currently capable of delegating commands to normal users, and requires some finesse when executing certain commands due to NFS permissions. Email admin-at-bx.psu.edu if something under sequencing/ doesn't look right.

In short, datasets are organized into "volumes" distributed between 2 servers and multiple NFS filesystems. 454 datasets are organized by one volume per month, and Illumina datasets are one volume per dataset. SSDFS fully abstracts the actual location of the volume (server/filesystem). If you need direct access to a volume, it will be under /afs/bx.psu.edu/depot/data/schuster_lab/.ssdfs/vol/by-name/<name>

Layout:

  • /afs/bx.psu.edu/depot/data/schuster_lab/sequencing/
    • 454/ (all 454 runs)
      • $YEAR/
        • $YEAR/$YEAR-$MONTH/
      • incoming/
    • illumina/ (all illumina runs)
      • $YEAR/
        • $YEAR/$RUN_NAME/
      • Instruments/
      • HWUSI-EAS610/
      • incoming/
        • illumina-4 -> /nfs/s2.persephone.bx.psu.edu/md1k-4-data/illumina/
        • illumina-5 -> /nfs/s2.persephone.bx.psu.edu/md1k-5-data/illumina/
        • illumina-6 -> /nfs/s2.persephone.bx.psu.edu/md1k-6-data/illumina/
    • scripts/ (Various shell scripts, qsub scripts, etc)
      • 454/
    • support/
      • 454/
      • illumina/

support/

  • /afs/bx.psu.edu/depot/data/schuster_lab/sequencing/support/

The sequencing support directory contains manuals, software, and reference genomes used during sequencing and sequence processing.

projects/

  • /afs/bx.psu.edu/depot/data/schuster_lab/projects

The projects directory contains directories for project specific data including the Tasmanian Devil, KB1, and Woolly Mammoth projects.

users/

  • /afs/bx.psu.edu/depot/data/schuster_lab/users

The users directory contains directories for people within the Schuster Lab.