UV-300 Supercomputer

Making sense of genomes at individual or cohort scale is achived by comparing individual or cohort genome data with existing knowledge bases or other cohorts. Existing knowledge bases that capture structural, functional, biomarker and pharmacological information are expanding rapidly e.g. UCSC Genome Annotations, ENCODE, GWAS catalog, PharmGKB,  and GO. Population databases now contain information across 1000s-10,000s of genomes e.g., 1000 Genomes project, Exome Sequencing Project, the UK10K consortium, and AmbryShare. Analysis takes many forms, including database queries, statistical approachs such as GWAS or machine learning techniques. Co-analyzing user data with the exploding volume of public/private datasets require large number of cores, GPUs or memory. 

A typical metagenomic analysis requires a) assemblies (genome or transcriptome), b) post-assembly analysis and validation such as searching against known bacterial and viral databases, classification of new assemblies, and quantitation and finally, c) correlation studies e.g. with metabolites. The memory requirements for de novo assembly increase dramatically with genome size. In a recent survey of memory efficient algorithms [Kleftogiannis D], investigators attempted to run a variety of assembler on Cloud or commodity servers. For large genomes or data sets, these runs fail or result in more contigs i.e. unfinished assemblies. Researchers from Oklahoma State University completed the largest metagenomics assembly to date by sequencing data from a soil metagenome that required 4TB of memory [Couger BM].

Sequence search algorithms e.g. BLAST can specially benefit from availability of large amount of RAM available on this appliance. In these search algorithms, newly assembled sequences are compared against large GB to TB sized databases of known DNA and protein sequences. Once the database files are loaded into RAM, they can be accessed in under a microsecond with shared memory – randomly reading anywhere. Empirically, in-memory data, on systems similar to this supercomputer, is 1,000x faster than data on hard drives.

Another aspect of metagenomic studies is human contamination removal from microbiome samples. Microbiome samples that are collected from saliva, gut, skin, nasal cavity and other sites are comprised of a substantial amount of human DNA. Typical microbiome studies collect many samples (sites or longitudinal). For example, it has been reported that some samples in The Human Microbiome Project contain up to 95% human sequence and 4% of the samples contain >10% human reads. Studies show repeatedly that de-identified human genome data can be re-identified [Naveed M]. So while there is no significant concern in current regulatory landscape, it may become an issue in coming years. Newer algorithmic approaches exist which are computationally expensive, that can be used to pre-process all microbiome samples to do a superior job of human contamination removal.

Supercomputer Configuration Summary

• High memory-to-processor ratio: Intel Xeon E7-8867 v4 with 24 CPUs/socket  
• SGI NUMALink™ 7 interconnect (NL7; 7.47GB/s bidirectional peak): Ultra low latency: All-to-All & Multi-dimensional All-to-all network topology
• Extreme I/O: 12 PCIe Gen3 slots per chassis
• NVidia Pascal GPUs
  • 5.3 TFLOPS of double-precision floating point (FP64) performance
  • 10.6 TFLOPS of single-precision floating point (FP32) performance
  • 21.2 TFLOPS of half-precision floating point (FP16) performance
  • NVLink high-speed interface provides GPU-to-GPU data transfers at up to 160 Gbps of bidirectional bandwidth
  • HBM2: offering three times (3x) the memory bandwidth of the Maxwell GM200 GPU
  • Unified Memory and Compute Preemption via CUDA 8

System Configuration Summary

This supercomputer is integrated into GBSC's computing cluster SCG and meets moderate risk compliance requirements along with dbGaP compliance requirements.  The supercomputer and the cluster nodes all have access to 7 petabytes of high-performance, redundant storage, divided across two storage subsystems (the older DDN and the newer Isilon).  Analyses are managed on the supercomputer and the other nodes via a Slurm job scheduler, which controls access to the resources provided by all the computing nodes.  By allowing access to only the resources needed for each job, the supercomputer can maximize its utilization and minimize its idle time.

Performance Benchmark

The SCGPM team has extensively benchmarked the UV300 for OPI/OpenMP and Cuda use cases. These results will be made public available shortly. In the meantime, please reach out to the administrators for details.

Deep learning for Learning Health System

Nigam Shah's research student Stephen Robert Pfohl uses patient data in the electronic health record (EHR) for the development of algorithms and models capable of automatically performing patient-level outcome prediction, risk stratification, and clinical decision support for the eventual goal of developing a learning health system. The use of EHR data poses challenges for these tasks in that it is heterogeneous and sparse with a complex temporal structure. For instance, a typical patient record contains multiple visits (potentially over several years) where each visit may contain coded data (e.g. diagnoses, procedure, and medication codes), unstructured text in clinical notes, laboratory measurements, linked genomic data, and densely sampled time-series or waveforms from bedside monitors. It is the case that models developed with EHR data often fail to generalize to new health systems and domains that differ from that which they were trained due to covariate shift and a lack of interoperability between sites.

Stephen leverages deep neural networks to develop methods capable of adapting learned representations of human disease to new domains with minimal input of human experts. The UV-300 NVIDIA Tesla P100 GPUs are well suited to deep learning applications. As this research involves the training of large and complex models on the basis of hundreds of millions of anonymized records from millions of patients, the high-performance capabilities of the UV-300 have greatly accelerated this research.