gridengine.info

{ crossposted to blog.bioteam.net and gridengine.info }

Amazon made a very important announcement today, releasing new EC2 server types and network configurations that significantly enhance the Amazon AWS environment for people who are interested in cluster computing, compute farming and high performance computing (HPC) on the cloud.

The announcement is here for those who are interested:

http://aws.typepad.com/aws/2010/07/the-new-amazon-ec2-instance-type-the-cluster-compute-instance.html

I'm thrilled that this news is now public, the service is up for use and I can finally start testing, blogging and benchmarking in the "real" production environment.

In the next few days I'll be blogging over on http://blog.bioteam.net, concentrating initially on seeing how storage and storage IO speeds differ on the new instance types. For life science types like myself, one of the biggest hassles in the cloud is due to the fact that we tend to be more performance bound by the speed of storage and file IO than anything else. The 10GbE networking changes and non-oversubscription of the network links along with the ability to group nodes together may mean very very interesting things are now much more feasible on the AWS platform.

Because I'm going to first concentrate on storage and IO stuff on the new offering I wanted to quickly show Grid Engine running on the new server types.

Even a single node SGE cluster can do reasonable work now as the cc1 instance type includes a pair of quad-core Nehalem CPUs along with ~23GB memory and a 10GbE ethernet backend.

We will be blogging and talking much more about how to use Chef Server to orchestrate self-assembling Grid Engine clusters and compute farms on this new service but since that may not happen until later -- I just wanted to throw up a teaser post showing SGE 6.2u5 running in single-node mode on the new HPC offerings from Amazon.

qstat output showing 16 CPUs (click for full-size):

qhost output showing system resources (click for full-size):

Offtopic but just wanted to post a short note that I'll be giving a short talk (~10) in NYC on May 28th at the 2009 Amazon AWS Start-Up Tour.

The tour details and dates are here:
http://aws.amazon.com/startupproject/

The 4 AWS user presentations on the 28th will be from

• Sam Lessin, CEO, drop.io
• Dan Gill, VP Business Development, Gotuit
• Chris Dagdigian, Founding Partner, BioTeam
• Brian Adams, Co-Founder and CTO, Admeld

Say 'hi' if you attend the event!

Not strictly on-topic for this blog but it is the main Amazon cloud feature that I've been anxiously awaiting for some time now.

Simply put, Amazon is now willing to accept delivery of eSATA and USB disks for large-scale data ingestion into the S3 storage cloud service. This is far faster than internet based methods, particularly if you are dealing with daily terabyte-scale data that you'd like to park in some sort of external utility service.

I think this may be a big deal for life science and can forsee quite a bit of scientific data making "1-way" trips into the cloud for long term storage and even secondary processing via cloud server instances. We'll see.

These links cover the just-launched service in detail:

• http://aws.typepad.com/aws/2009/05/send-us-that-data.html
• http://aws.amazon.com/importexport/

Sun has published a new flash video demo (registration required for viewing) here: http://www.sun.com/offers/details/sun_grid_engine.html

Description:

Sun envisions a world of many clouds, both public and private, that are open, compatible, and designed for all types of applications—including high performance computing. Sun is extending cloud computing to High Performance, building not just a cloud, but an entire Cloud eco-system. Through this narrated demonstration, learn how Sun Grid Engine fulfills the promise of cloud computing in HPC.

What does this have to do with Grid Engine? Well the UniCluster Express suite includes Grid Engine with integrated monitoring and reporting tools.

I did a project for Univa UD where I documented the steps required to get their product integrated into the Amazon EC2 elastic compute cloud. That is the whitepaper referred to in the marketing materials below.

I've agreed to participate in a webinar and answer questions about the project, click on through for the full event details.