We have been silent for a while – not for the lack of something to say but lack of time to say it in ;-). But today is a very special day: yesterday was the first anniversary of the day Chameleon, a cloud computing experimental instrument project that Nimbus team is proud to lead, went public. Considering how busy we are, breaking the silence is a bit of a treat but after all that’s what anniversaries are for!

We just reduced image propagation time on FutureGrid’s Sierra cloud at UCSD from hours to minutes! This magic comes courtesy of Nimbus LANTorrent.

We blogged about LANTorrent before: it can distribute the same file among many nodes using peer-to-peer techniques. It is available in Nimbus since version 2.6 and allows users to efficiently deploy a cluster of virtual machines based on the same image. Installing and configuring LANTorrent on the Nimbus nodes (both service and hypervisor nodes) is easy; it took only a couple of hours on Sierra (all the details are explained in the LANTorrent Configuration section of the Nimbus documentation).

A few weeks ago (May 23rd-26th) I traveled to sunny Newport Beach, California to present a paper, Improving Utilization of Infrastructure Clouds, at CCGrid 2011. Our paper addresses one of the main challenges faced by infrastructure cloud providers: ensuring that resources are utilized efficiently while still providing resources on-demand. To solve this catch-22, we deployed backfill VMs on idle VMM nodes. For evaluation, we deployed Condor in the backfill VMs and demonstrated an increase to 100% utilization of the infrastructure resources. All of the details are in the paper, so I won’t elaborate here. You can also try backfill for yourself with Nimbus 2.7.

I had the privilege of presenting Cumulus: Open Source Storage Cloud for  Science at the Science Cloud 2011 workshop yesterday.  While I was focused on our open source S3 implementation ideal for the extensibility and scientific experimentation, many other interesting topics were presented. Shane Canon present a very interesting look at common misconceptions about the cloud in scientific circles.  In it he exposed some truths about what ‘on demand’ ultimately means to a data center.  He worked to illustrate where on the hype curve the cloud currently is, and what features work for science and what was missing.  Elasticity for bursty applications is a clear win but a sighted glaring gap is the lack of a shared file system.  A shared file system is an assumed service to most scientific users coming from the grid and most other HPC platforms.  This need for a shared file system struck a chord with me and it seemed to be a common theme at the workshop.  Lavanya Ramakrishnan gave a talk on Magellan: Experiences from a Science Cloud.  In it she mentioned the struggles scientific users had with their applications inside of VMs.  One was the difficulty staging in data into the VM’s space.  A couple of other talks discussed the huge volumes of data created by scientific applications. All of this discussion made me wonder if a Cloud agnostic shared file system service could be created and if such a thing could solve these problems.

When we think of science we don’t immediately think of quality assurance and yet… scientists have to run their codes somewhere, they need hardware, the hardware needs software, and the software needs to be operated reliably and efficiently – enter the Quality Assurance (QA) team of TeraGrid: the most powerful open science resource.

Shava Smallen, the co-lead of the TeraGrid QA team, told me recently of their first venture into infrastructure clouds. The TeraGrid Science Gateway projects have been experiencing scalability problems with grid infrastructure. A potential solution came out in the form of GRAM 5; the scientists developed scalability tests to see if it solved their problem — but where could they run them? They tried Ranger, a top-of-the-line TeraGrid resource at Texas Advanced Computing Center (TACC). But Ranger is a powerful resource, very much in demand for large scientific computations that cannot run elsewhere — and thus the QA team found itself with tests all ready to run – but no resources to run them on.

Cloud computing users think on-demand availability is the best thing since sliced bread: it enables elastic computing, outsourcing for applications requiring urgent or interactive response, and reduces wait times in batch queues. But if you are a cloud provider you might not think so… In order to ensure on-demand availability you  need to overprovision: keep a lot of nodes idle so that they can be used to service an on-demand request, which could come at any time. This means low utilization. The only way to improve it is to keep fewer nodes idle. But this means rejecting more requests – at which point you’re not really on-demand… a veritable catch-22.

I often get asked if there is any published work evaluating performance and cost of scientific applications on IaaS clouds and comparing them to using clusters — and I always say LOTS! …and then can’t remember more than a few off the top of my head ;-). So I recently put together a list — included below — of various evaluation and comparison efforts I’ve been able to find. They  look all sorts of aspects of performance — from low-level benchmarks to applications of various types, from reliability to cost. They all tend to focus on somewhat different aspects of the issue and collectively paint a picture blessings and challenges of cloud computing for science.

BaBar from the children’s books is a young elephant who comes to a big city, and brings back the benefits of civilization to other elephants in the jungle. He also happens to be a very apt mascot for a high-energy physics project.

The name BaBar actually derives from the B/B-bar subatomic particles produced at the SLAC collider in Stanford, California during electron-positron collisions. These experiments help us achieve a better understanding of the relationship between matter and anti-matter and ultimately answer questions about the nature of the universe. This groundbreaking research is moving forward at a brisk pace: the BaBar scientists have petabytes of data, a plethora of insight, and a recent Nobel prize to show for it.

HPC in the Cloud posted a nice article describing how scientists from the Canadian CANFAR projects are using cloud computing to deal with their data problem. Quoting from a recent white paper by Nicholas Ball and David Schade:

“in the past two decades, astronomy has gone from being starved for data to being flooded by it. This onslaught has now reached the stage where the exploitation of these data has become a named discipline in its own right”

Happy New Year!

To get it off to a new start check out the call for papers for the ScienceCloud 2010 workshop – announced right before Christmas!

The last year’s Science Cloud workshop was a great venue for anybody interested in cloud computing for science. The program covered everything from scientific cloud platforms (and how to set them up), through standards and middleware, to case studies of scientific applications on commercial cloud platforms such as Amazon and Azure. The latter were perhaps the most interesting of the workshop – and in fact one of them, a performance study of a cosmology application on Amazon from Lawrence Berkeley National Lab won the best paper award. The slides and papers can be viewed online – still a great reference to see what’s happening in cloud computing for science.  Some of the papers will also appear in the Scientific Programming Journal’s special issue on science-driven cloud computing.