“I have used the new Gemini Observatory Archive to retrieve data for our
LP-3 and I found it very easy to use and fast so I don’t anticipate any problems. I am
grateful to the team at Gemini that has been working on it. “
— Catherine Huitson, PI of GN-2015B-LP-3
To solve all these problems, I initiated the
FITS (Flexible Image Transport System) storage project and wrote a software package
that would scan through the FITS files on a file
system and record details of them in a database. A simple web interface presented concise summaries of the data files. Gemini also
bought some modern LTO4 (high capacity
and performance) tape drives, and wrote software to record (in the database) file details as
it wrote them to tape.
With these changes, we’d never lose a single
file again. We’d also be able to know with a
few mouse clicks which files were safely on
tape, and which tapes they were on. We wrote
scripts that tied into the system to clear old
data off disk (after first checking it was safely
on multiple tapes) to ensure that disks didn’t
fill up.
Fast forwarding into the Transition Plan years,
we realized that the FITS Storage System had
evolved to the point where it would be possible, with a reasonable amount of effort, to
expand it into a fully fledged archive system.
As neither the Hilo nor La Serena base facilities have adequate internet connectivity for
users to simultaneously download data from
all over the world, they seemed unattractive
as locations to host an archive server. However, when we researched the costs of Cloud
computing services — such as Amazon Web
Services and Google Cloud Platform — it became pleasantly apparent that substantial
cost savings could be realized by moving to
an in-house developed archive system hosted
on a commercial Cloud platform.
We were given the go-ahead at the end of
2013, spent 2014 developing a prototype sys-
January 2016
tem; then, following approval by an external
group of testers, worked through 2015 transforming the prototype into the fully developed GOA. The system is now deployed on
Amazon Web Services, using an EC2 (Elastic
Compute Cloud) virtual server and storing
data on S3 (Simple Storage Solution) with
backups on Amazon Glacier.
Cloud Computing
The decision to store data “in the Cloud” has
raised more than a few eyebrows, but once
we started researching this option, it quickly
became obvious that this approach offers
many advantages. As far as we know, we’re
the first major observatory hosting our archive on a commercial Cloud computing platform, though I’m sure more v