Today we made open source Infrastructure-as-a-Service capabilities for science just a little bit better… The new Nimbus 2.6 makes your images get to the nodes a lot faster and provides capabilities allowing administrators to easily and dynamically shift resources between their tried-and-true batch scheduled cluster and a cloud — depending on where they need them more. The Nimbus Context Broker gets shiny new REST-based interfaces, which opens it up to new applications.
The highlight of the release is unquestionably the introduction of the fast propagation LANTorrent tool. The deployment of many identical virtual machine images represents a very common pattern in scientific use of clouds. It occurs when researchers deploy a virtual cluster, augment an existing cluster by adding more worker nodes, or enlarge a set of platforms for a volunteer computing application.
Tests of LANTorrent on Magellan by John Bresnahan; we distribute a thousand 1.2 GB images in 10 minutes.
LANTorrent leverages this pattern to optimize image distribution within your cloud. Built on the ideas underlying BitTorrent, LANTorrent uses streaming to propagate the VM image file to hundreds or thousands of nodes. Unlike copying, streaming does not wait for the file to fully arrive at its destination before forwarding it on – by the time the end of the file reaches the first node, its beginning may have been forwarded to many other nodes. The only limiting factor to how many streams we can have “in flight” at any given time is the capacity of the LAN switch. And this is where the “LAN” part of LANTorrent comes in – it tries to optimize the number of transfers in order to saturate the switch without causing contention. In addition to that, LANTorrent detects all same-image transfers taking place within a time bucket – even if they are part of different requests – and includes them in existing streams. As shown on the enclosed graph, on the ANL’s Magellan cloud we were able to deliver a thousand 1.2 GB VM images within 10 minutes. Moreover, the graph also shows that LANTorrent can scale better than linear (the red line in the graph) thanks to multiple transfer detection.
Another significant addition in this release is the dynamic cloud configuration management tool. Adding or removing nodes from a Nimbus cloud used to be complex: you had to take the cloud down, manually adjust various files, then restart your cloud. This new feature makes all that a thing of the past – now you simply boot the node you want to add (configured with Nimbus), run the nimbus-nodes program and get back to enjoying your coffee. Most importantly the adjustment takes place on-the-fly, while the cloud is running – no downtime, and no inconvenience to your users. This dynamic adjustment feature paves the way for future improvements that will allow administrators to easily and dynamically move resources between their say, PBS cluster and a cloud as we plan to do in FutureGrid. A host of new upgrade tools accompanying this new addition mean that your Nimbus cloud is easier to manage than ever.
Finally, last but not least, the Nimbus Context Broker — successfully used to deploy turnkey virtual clusters for scientific production runs — has a new interface, based on HTTP/REST, in addition to WSRF. This makes it much easier to use the Context Broker in new integrations — in particular ones that rely on different languages – and has been a longstanding user request.
If you are looking for more information about our new features, take a look at the changelog, lots of new things to explore. And in case you are looking for something to do, we will be announcing feature lists for 2.7 soon ;-).
Recent Comments