High Performance Compute on AWS , Google & Azure
Joey Jablonski
In 2009 I wrote a blog questioning the validity of High Performance Compute ( HPC ) workloads running in the cloud . At the time , the technology in the public cloud was simply not ready to accommodate the types of workloads and communication patterns seen in the HPC space . We have come a long way since 2009 . Today we are at the point where we not only have the technology in the cloud needed to run HPC workloads , but also have real-world use cases of companies who are successfully doing it .
The Broad institute is using Google Cloud and HPC to process large-scale genomic information to accelerate scientific progress in cancer , diabetes , psychiatric disorders , and many other diseases .
A global designer and manufacturer of automatic test equipment uses cloud-based HPC to compliment on-premise resources to ensure integrated circuit designs are reliable and delivered on schedule .
A global financial services firm is leveraging cloud HPC resources for portfolio analysis , risk modeling and compliance activities to stay ahead of changing regulations .
While there are still specific categories of HPC workloads that are not as efficient or performant in the cloud , such as Electronic Design Automation ( EDA ) and shared memory problems , many workloads for genomics , geospatial , rendering and image analysis are finding a home in public clouds . The dominant cloud vendors , including AWS , Azure and Google , have invested to ensure that workloads are easy to migrate , manage and scale . They have been complemented by companies like Cycle Computing and Univa which have added features to easily extend HPC environments from the data center into the cloud , minimizing disruption to developers and users .
HPC workloads will continue to grow in the cloud , as HPC communities continue to adopt more advanced technologies . One ongoing area of enhancement is with application vendors who have traditionally run in the data center , but are investing in optimizing their applications to run in an elastic environment . One company that is leading the EDA space is Cadence , a firm that keeps evaluating and investing in software that is efficient in the cloud .
Parallel file systems are a key element of most HPC deployments . Intel leads the way with Lustre and continues to invest in development and tuning to ensure that it performs well in the cloud and that customers have easy to use templates to deploy it .
There will always be HPC workloads that demand specialty hardware and connectivity from Cray and Mellanox which will be difficult , if not impossible , to run in the cloud . But we should still leverage the cloud for the rest of the workloads that operate easily and scale seamlessly there . When evaluating your HPC environment for growth , technology transition and capabilities , look to the cloud for new innovations and capabilities that will enable your organization to be more flexible and scalable .
74 | THE DOPPLER | SUMMER 2016