HPC in the cloud

Cloud Play


Cloud computing is most definitely here, but does it have a role in HPC? We discuss changes in HPC that could be solved effectively by cloud computing.

I'm not a big Bob Dylan fan, but when it comes to HPC, "The Times They Are a-Changin'." Some of this change is going to the cloud, not because it would be really cool, but rather because HPC has existing workloads that fit well into cloud computing and can save money over traditional solutions. Perhaps more importantly, I think HPC has evolved to include non-traditional workloads and is adapting to meet those workloads – in many cases using cloud computing to do so. I will explain by giving two examples.

1. Massively Concurrent Runs

At one HPC center that I'm familiar with, users periodically submit 25,000 to 30,000 jobs as part of a parameter sweep; that is, they run the same application with 25,000 to 30,000 different data sets. Many times the application is a Matlab script, a Python script, an R script, a Perl script, or something particularly serial (i.e., it runs well on a single core). The same script is run but with thousands of different input files, resulting in the need to run thousands of jobs at the same time. Many times these applications run fairly quickly – perhaps in a couple of minutes  – and many times they do not produce a great deal of data.

A closely related set of researchers study operating systems and security, during which they run different simulations with different inputs. For example, they might run 20,000 instances of an OS, primarily a kernel, and explore exploits against that OS. As with the previous set of researchers, the goal is to run a huge number of simulations as quickly as possible to find new ideas about how to protect an OS and kernel. The run times are not very long, but they must run the tests against a single OS. Consequently, they run thousands of jobs at the same time and then look through the results and continue with their research.


Use Express-Checkout link below to read the full article (PDF).

Read full article as PDF:

Price $2.95

Related content

  • Gartner: IBM’s Blue Cloud Needs to Mature

    IT market researchers Gartner recommend that corporations view IBM’s “Blue Cloud” portfolio as a first step towards cloud computing but as no more.

  • StarCluster

    Cloud computing has become a viable option for highperformance computing. In this article, we discuss the use case for cloud-based HPC, introduce the StarCluster toolkit, and show how to build a custom machine image for compute nodes.

  • News
  • New Community Manager Helps Cloud.com Move into Open Source

    "The LAMP stack helped build the web; open source helped build the web; and Cloud.com wants to shape the future of cloud computing," says Mark Hinkle, Vice President of Community at Cloud.com.

  • Linux News

    Cloud News
    • CloudStack moves toApache
    • Red Hat Data Grid 6 beta released
    • News Bites

    Kubuntu Sponsored
    • Blue Systems sponsors Kubuntu
    • IBM PowerLinux Solutions
    • Spyrus Pocket Drive

    Ubuntu News
    • Ubuntu 12.04 LTS for desktop launched
    • Ubuntu 12.10 flies with Quantal Quetzal
    • Coverity 6.0 released

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95