Object Store for Large-scale Data Sharing

With the ever-increasing amounts of scientific data that are being generated and that are required for discovery, more and more public data repositories are being established, and these data sources have become very valuable for many investigations.  These sites gather together many different kinds of data about a particular domain and allow other researchers to access this information for their work.  

What makes a data repository valuable is the metadata that is delivered with the files:  data about the data.  This metadata is critical to helping the downloading researcher decide if the data that is presented is relevant to the investigation that they are conducting, especially if that investigation is assembling data from multiple sources.  For a FASTQ file from a sequencing run, the metadata referring to it might be the type of sequencer used, the date and time of the run, the read lengths performed, the organism sampled, the chemical concentrations measured, and the institution where the run was performed.

Sometimes the metadata is generated after the fact from the underlying data file.  This post-upload metadata can be added by manual curators, who examine the file and its metadata to determine its self-consistency.  Curators often standardize the metadata associated with a file onto a specific ontological vocabulary, to make searches through the database more consistent.  Derived metadata can also be added to the record by automated processes which assess each new piece of data against quality control metrics.

The challenge that preparing such a data repository presents is that the data and metadata need to be coordinated with each other, but are usually stored in separate places.  The files with data are typically stored in a hierarchical directory structure, while the metadata often resides in a database, with pointers into the data file directories to associate the metadata with its data.  

The object store solves this problem by storing the metadata with the data files themselves.  An object in an object store can have a large amount of key-value metadata associated with it, along with the data in a file.

The figure above shows schematically what this use case looks like.

In this diagram, the Stanford researchers place the data they want to share publicly in objects in the object store.  These objects contain the data that they are sharing together with metadata about that data.  For example, if the data were FASTQ files from a sequencing run, the metadata would be information like read length, the species of the sample, and reagent concentrations.  The object store would then share these objects with “Other investigators” who need this data for their work, via normal, unauthenticated HTTP access.  The URLs for the objects are generated automatically, so no effort needs to go into setting this scenario up, other than declaring the objects as world-readable.

Because of the multidisciplinary nature of many areas of biomedical science and the number of resources required to tackle large projects, many scientific investigations are conducted between labs, between institutions, and/or within consortia.  Examples of collaborative efforts between institutions within consortia are the Center for Excellence in Stem Cell Genomics, the Undiagnosed Diseases Network, the ENCODE project, and the ClinGen Resource.  To work efficiently together, these disparate groups typically need to share files and datasets, but to maintain data security, this sharing must occur exclusively between the member organizations, and not with the Internet at large.

The options for setting up this type of file sharing all have significant downsides.  The easiest thing to do is to give one of the collaborators login access to the servers of the other, but this method gives the outside collaborator access to more of the sharing collaborators’ systems than the situation calls for, and, in the case of UNIX machines, setting the group permissions on the shared files to include the outside collaborators can be very clumsy, since every file can be in only one group.  One side can set up a high-speed sharing server like Globus or Aspera, but these solutions are hard to configure and, in the case of Aspera, very expensive.  One collaborator can upload the shared data to the cloud for the other to download, but that download by the receiving collaborator will incur charges to the transmitting one, on top of the storage charges for the data while it waits to be downloaded.

The object store provides a solution to the problem of collaboration sharing.  The ability to apply fine-grained access control to the objects shared means that it is easy to configure the permissions so that only the needed individuals can download the objects shared.  The objects can be exposed as URLs, so they can be downloaded by any browser, without requiring any special client software.  And, in the case where an authenticated download is inconvenient, the object store can generate time-limited URLs which are active for only a few hours.

The figure above displays this use case in a schematic.

In this scenario, when Stanford researchers want to share data with the team that they are collaborating with, they upload it to objects in the object store, and set the sharing permissions to the members of that team.  Those collaborators can then authenticate themselves to the object store’s HTTP server, and download the data.  Other investigators outside of the consortium who may have the URLs for the data shared in this way will NOT be able to access the data.