gridengine.info

{ crossposted to blog.bioteam.net and gridengine.info }

Amazon made a very important announcement today, releasing new EC2 server types and network configurations that significantly enhance the Amazon AWS environment for people who are interested in cluster computing, compute farming and high performance computing (HPC) on the cloud.

The announcement is here for those who are interested:

http://aws.typepad.com/aws/2010/07/the-new-amazon-ec2-instance-type-the-cluster-compute-instance.html

I'm thrilled that this news is now public, the service is up for use and I can finally start testing, blogging and benchmarking in the "real" production environment.

In the next few days I'll be blogging over on http://blog.bioteam.net, concentrating initially on seeing how storage and storage IO speeds differ on the new instance types. For life science types like myself, one of the biggest hassles in the cloud is due to the fact that we tend to be more performance bound by the speed of storage and file IO than anything else. The 10GbE networking changes and non-oversubscription of the network links along with the ability to group nodes together may mean very very interesting things are now much more feasible on the AWS platform.

Because I'm going to first concentrate on storage and IO stuff on the new offering I wanted to quickly show Grid Engine running on the new server types.

Even a single node SGE cluster can do reasonable work now as the cc1 instance type includes a pair of quad-core Nehalem CPUs along with ~23GB memory and a 10GbE ethernet backend.

We will be blogging and talking much more about how to use Chef Server to orchestrate self-assembling Grid Engine clusters and compute farms on this new service but since that may not happen until later -- I just wanted to throw up a teaser post showing SGE 6.2u5 running in single-node mode on the new HPC offerings from Amazon.

qstat output showing 16 CPUs (click for full-size):

qhost output showing system resources (click for full-size):

This is a bugfix/maintenance release, read the full announcement here. .

As always, checking the list of fixed bugs and issues is a good way to start deciding if an upgraded is needed and how urgent it may be.

Photos from the 2009 Grid Engine Workshop in Germany. Please help me name and tag the participants!

Also, my talk on "Grid Engine & Amazon Compute Cloud" is online here:
http://blog.bioteam.net/2009/09/09/grid-engine-amazon-ec2/. The other talks and training slides are slowly making their way onto the net. We should have video from all the talks as well.

The 2009 Sun HPC Workshop started today with some tutorial sessions. Dan T and I are running a Grid Engine Administration workshop right next door to the Lustre folks.

Important Note: Sun has changed the license terms for this release. The full release from Sun.com can only be used for 90 days for free. The courtesy binaries are still free for all to use but the distribution will not include the Amazon EC2 cloud adaptor or the excellent "sgeinspect" tool. Source code for both of these components is available under the SISSL license so theoretically community members can build versions for themselves.

The full release announcement is here:
http://gridengine.sunsource.net/news/SGE62u3-announce.html

For me, the most important new features are the SGEInspect tool (screenshots of which you can view online at http://www.flickr.com/photos/chrisdag/sets/72157617805352910/ and the exclusive host scheduling feature which now removes the need for PE-based 'hacks' to achieve the same goal.

The license change is interesting, I need to see how hard it is to build sgeinspect from source code, it really is a powerful new tool. It's a shame that this won't be part of the free distribution but then again I want Sun to make product and support revenue off of SGE so I can see the point.

2009 Sun HPC Workshop
September 7-10, 2009
Regensburg, Germany

The SGE Workshop returns to Regensburg, now operating as a conference track within the 2009 Sun HPC Workshop event.

Go here for details & registration:
http://hpcworkshop.com/

Pictures I took at the 2007 event:

• Regensburg 2007 Pictures
• 2007 Workshop Pictures

A few weeks back I recorded a large and clear screencast of my most recent laptop install using the SGE GUI and then processed the heck out of it so that it is a decent quicktime movie weighing in at only about 3.1MB for download.

Not major news but may be of interest to people who have not seen the new graphical installer in action yet. Of particular note the things that I like about the SGE GUI install methods:

• Wildcards and ranges for hostnames (not possible in other SGE install methods)
• Does remote installs when passwordless SSH is available
• When things fail it actually collects decent log and debug information in a nice central location
• Does some pre-install sanity checking and warns of potential issues
• The final summary of your install with options to print or save the key details is a fantastic new addition

Click on the image to start the download:

Sun is hosting an event along side the international supercomputing conference: http://www.supercomp.de/isc09/. The Sun event schedule does not heavily mention SGE but the Sun HPC leadership team will be present (along with some SGE developers I'm guessing) and there are sessions covering grid middleware and Univa's UniCluster product which includes SGE.

What: 2009 Sun HPC Consortium
Agenda: http://events-at-sun.com/hpc-hamburg09/agenda.php
When: June 21 - 22, 2009
Where: Hotel... Le Royal Méridien Hamburg (Starwood Hotel)
An der Alster 52 - 56
20099 Hamburg, Germany
Phone: (49)(40) 2100 0
Fax: (49)(40) 2100 1111
Registration:
http://events-at-sun.com/hpc-hamburg09/registration.php


Offtopic but just wanted to post a short note that I'll be giving a short talk (~10) in NYC on May 28th at the 2009 Amazon AWS Start-Up Tour.

The tour details and dates are here:
http://aws.amazon.com/startupproject/

The 4 AWS user presentations on the 28th will be from

• Sam Lessin, CEO, drop.io
• Dan Gill, VP Business Development, Gotuit
• Chris Dagdigian, Founding Partner, BioTeam
• Brian Adams, Co-Founder and CTO, Admeld

Say 'hi' if you attend the event!