gridengine.info :

Site update

dag@sonsorol.org (chris) — Tue, 22 Jul 2008 11:21:44 -0400

If you are reading this and seeing a peach colored background then you are seeing the new default Typo 5.1.1 theme. I updated the RAILS and Typo installation on this box today and decided to at least experiment with the new theme for a bit. The article area is wider which I like (and I never got back into the Scribbish 2.0 theme to make that wider...) so I may take the lazy way out and stick with this theme.

Leave a comment if you like this theme or if you'd prefer the white-background "Scribbish 2.0" theme. Thanks!

Collection of usable ARCo queries

dag@sonsorol.org (chris) — Tue, 22 Jul 2008 09:03:13 -0400

Thanks to Jana, there is a new wiki page up at http://wiki.gridengine.info/wiki/index.php/ARCoQueries that documents some additional contributed ARCo queries for the Grid Engine Analysis & Reporting Console.

Why upgrade? DanT explains SGE from 5.x through 6.2 and beyond

dag@sonsorol.org (chris) — Fri, 18 Jul 2008 14:58:52 -0400

Dan has posted a great overview of how Grid Engine has changed since the version 5.x days, couched in the context of answering the "Why should I upgrade SGE?" questions that often come up.

I won't even excerpt it, the full article is well worth a read:
http://blogs.sun.com/templedf/entry/why_upgrade

Univa UD UniCluster Express (and Amazon EC2)

dag@sonsorol.org (chris) — Mon, 14 Jul 2008 19:55:25 -0400

Univa is a company that I became familiar with when Steve Tuecke gave a talk at the 2007 Grid Engine workshop in Regensberg, Germany. Steve's talk was called "Univa Open Source HPC Cluster and Grid Software" and you can download the PDF version here.

Steve's talk and his company were pretty interesting and he specifically hit on some areas I had mentioned in my talk on "LSF vs. SGE" where we discussed the lack of ISV and commercial entities capable of supporting the Grid Engine community with service, support, consulting, training and other "value added" activities. I came home from Germany thinking that Univa would be worth watching (also Steve had dropped some hints informally about 'some news may be happening soon...') -- and sure enough a short time later Univa became "Univa UD" as Univa and United Devices became a single company.

If you point a web browser over at http://grid.org or the corporate site at http://www.univaud.com you'll quickly see that they have a product out called "UniCluster Express".

Cluster stacks are normally not all that exciting to me but Univa is doing something different that makes it worth checking out ...

Click on through for the full article ...

The UniCluster Express stack is free, 100% open-source and contains the following core components: Grid Engine, Grid Engine Analysis & Reporting Console (ARCo), Ganglia, Globus and a custom Java based central monitoring console application.

But that is not the really interesting part. The eye catching bit is that Univa UD is offering free support for this integrated stack.

Yep.

Free support for an integrated stack including some really complex bits like SGE ARCo and the impossible-for-most-mortals Globus stack (heck, I attended a GGF meeting and actually sat through the Globus tutorial and still could barely get it functioning in my lab environment...)

Univa UD has taken already popular open-source compute farming technologies, added in a new Java based monitoring console and bundled everything up into a installable tarball. Free support comes through grid.org and I'm guessing that they plan to actually make money by offering customization, commercial support and other services to people willing to pay.

I recently took the software for a spin myself and got it integrated into the Amazon Web Services EC2 elastic compute cloud framework. My work was written up as a HOWTO whitepaper and was documented in a series of recorded screencasts.

One of the screencasts is linked to at the top of the article, all of the others are available by following this URL:
http://www.screencast.com/users/BioTeam/folders/UniCluster-in-Amazon-EC2

If you want the PDF Whitepaper you'll have to get it via a web form over at http://www.univaud.com/hpc/wp-unicluster-amazon-ec2.php.

Disclaimer/Disclosure: In my role as part-owner/employee of BioTeam, my company earned money on this project. Univa UD basically challenged us to get their stuff working in EC2 and offered to pay for a few days of consulting time. A more detailed disclaimer can be found within the whitepaper.

SGE and MPICH2 On Windows/Linux Heterogenous Systems

dag@sonsorol.org (chris) — Mon, 14 Jul 2008 19:03:29 -0400

Thanks to Jacek Strzelczyk for the new Wiki page entitled "Install and configure Grid Engine in heterogenic environment on Linux and Windows with MPICH2" that was posted earlier this week.

Feedback needed: Obsolete options and parameters considered for removal

dag@sonsorol.org (chris) — Tue, 24 Jun 2008 08:22:41 -0400

Grid Engine developers posted a list today of SGE configuration parameters and client arguments that are being considered for removal from the product because they are either obsolete or they duplicate settings found elsewhere.

The developers are seeking feedback and comments on their plans - if you have any please drop a line to the users@gridengine.sunsource.net mailing list. The current roadmap calls for these methods to be marked as 'deprecated' in the SGE 6.2 release with total removal planned for a future post-6.2 release.

The message can be found here:
http://gridengine.sunsource.net/servlets/ReadMsg?list=users&msgNo=25045

The full list of items being considered for removal can also be found after the jump ...

The parameters planned to obsolete are:

host_conf(5)
- processors
   obsolete, same as num_proc from the complex list

sched_conf(5)
- algorithm
   just default is allowed, no additional algorithms are planed
- params JC_FILTER
   huge performance impact plus may lead to wrong scheduling decisions

sge_conf(5)
- reprioritize
   redundant because hard bound to reprioritize_interval in sched_conf(5)
- shell_start_mode
   obsolete, value from queue_conf(5) is used
- set_token_cmd
   no known AFS support
- pag_cmd
   no known AFS support
- token_extend_time
   no known AFS support
- qmaster_params DISABLE_AUTO_RESCHEDULING
   equivalent to default reschedule_unknown=0:0:0
- qmaster_params merge ACCT_RESERVED_USAGE and SHARETREE_RESERVED_USAGE
   We can't imaging a use case to have these values separated
- finished_jobs
   qstat -j does not work with successful finished jobs. Code seems
   to work only with jobs going into error state.

user(5)
- delete_time
   change to internal, not changeable/visible field
   Implicit set by auto_user_delete_time

qconf(1)
- sep option
   obsolete, same as num_proc
- ks option
   obsolete, same as -kt scheduler

qmod(1)
- c option
   depreciated, use -cj or -cq
- r option
   depreciated, use -rj or -rq
- s option
   depreciated, use -sj or -sq
- us option
   depreciated, use -usj or -usq

SGE 6.2 beta 2 is out

dag@sonsorol.org (chris) — Thu, 19 Jun 2008 07:04:12 -0400

6.2b2 came out yesterday:

http://gridengine.sunsource.net/news/GE62beta2-announce.html

The list of bug fixes made since SGE 6.2 Beta 1 is online at http://gridengine.sunsource.net/project/gridengine/62patches.txt.

This is the latest beta release of SGE 6.2 and we really need more eyeballs and testers on this release to flesh out any remaining issues before 6.2 goes officially out the door.

There are some differences in 6.2 both in the install procedure as well as the daemons (sge_schedd is gone! -- It's now a thread within sge_qmaster). I posted a screencast recording of the SGE 6.2 Beta 1 installation a while back: http://gridengine.info/articles/2008/05/16/screencast-live-install-of-sge6-2-beta for those that may be interested in watching what the new install process looks like.

How to run Dytran applications under Grid Engine

dag@sonsorol.org (chris) — Wed, 28 May 2008 11:44:29 -0400

Gerhard Venter asked the users list for assistance in getting Dytran to run under Grid Engine. Once his issues were resolved, Gerhard was kind enough to write up a Wiki Entry on Dytran/SGE integration.

The wiki page is here:
http://wiki.gridengine.info/wiki/index.php/Dytran

Thanks Gerhard!

Dan reviews Grid Engine queues

dag@sonsorol.org (chris) — Wed, 28 May 2008 11:33:34 -0400

Dan is back and blogging up a storm, he recently posted a nice overview of Grid Engine queue basics. An excerpt is here:

... So, aside from governing the number of free slots on a host, what does a queue do? It controls the execution context of jobs that run in it. It determines what parallel environments are available, what file, memory, and CPU time limits should be applied, how the job should be started, stopped, suspended, and resumed, what the job's process' nice value is, etc. ...

Read the full article here:
http://blogs.sun.com/templedf/entry/intro_to_grid_engine_queues

Java DRMAA binding via JavaScript

dag@sonsorol.org (chris) — Wed, 28 May 2008 11:27:59 -0400

%  jrunscript -cp $SGE_ROOT/lib/drmaa.jar -f drmaa.js

Job 2 submitted
Job 2 has ended
Job terminated abnormally
%

Richard Hierlmeier seems to have joined the ranks of Sun Bloggers and has a facinating post up documenting how he used the javascript engine that ships with Java 6 to bind to drmaa.jar.

The post is here:

http://blogs.sun.com/rhierlmeier/entry/java_drmaa_binding_with_javascript

Creating Hadoop PE under Grid Engine

dag@sonsorol.org (chris) — Fri, 23 May 2008 10:13:24 -0400

Dan has found a great Sun blog article by Ravi Chandra Nallan post on integrating Hadoop into SGE via the use of a parallel environment.

Image source: http://hadoop.apache.org/core/

Links:

June 2008 SGE Workshops

dag@sonsorol.org (chris) — Fri, 23 May 2008 09:53:49 -0400

Consider this post a plug for the upcoming June 2008 SGE User and SGE Admin workshops that are being held in the Boston, MA USA area.

More details here:
http://blog.bioteam.net/2008/03/22/sge-training/

OSGC Presenter Slides

dag@sonsorol.org (chris) — Mon, 19 May 2008 11:40:17 -0400

The full list of talks along with links to PDF, PPT and video files can be found here:
http://opensourcegridcluster.org/programming.html

At the time I'm writing this, my PDF slides have not made it up on the official site yet. For anyone interested in those, here are some temporary download links courtesy of the Bioteam Blog. For people who saw me at the German SGE Workshop, the only new topic is "SGE & Amazon EC2".

Here is a short culled list of talk names and slides that may be of interest to this community:

Sun/Community State of the Union - Speaker: Fritz Ferstl
New features in Sun Grid Engine 6.2 - Speaker: Lubomír Petrík
Using OGF Standards for Grid and HPC - Speaker: Chris Smith
Grid Engine at the Texas Advance Computing Center - Speaker: Roland Dittel
Grid Heating: Dynamic Thermal Allocation via Grid Engine Tools (ppt) - Speaker: Paul Brenner
Service Domain Manager – Basics and Concepts - Speakers: Richard Hierlmeier & Ryszard Macidlowski
Accounting and Reporting Console Multi-Cluster Support - Speaker: Jana Olivova
Making Grid Engine Highly Available with Open High Availability Cluster and OpenSolaris - Speakers: Ashutosh Tripathi
HPC Visualization on the Grid - Speaker: Linda Fellingham & Dean Stanton
PluS: An Advance Reservation plug in for Sun Grid Engine - Speaker: Hidemoto Nakada
Berkeley Laboratory Checkpoint Restart - Speaker: Eric Roman

Screencast: live install of SGE6.2 beta

dag@sonsorol.org (chris) — Fri, 16 May 2008 15:55:53 -0400

Truthfully speaking, after taking an overnight flight back to Boston from California I really was in poor shape to actually get any real work done today.

I've recorded my experience installing the fresh release of SGE 6.2beta on my laptop. The video screencast itself is hosted over at a BioTeam site -- it's only fair because BioTeam is paying for the hosting costs as well as the screencast recording software!

The video screencast is linked off of this blog post:
http://blog.bioteam.net/2008/05/16/sge-62beta-unboxing-screencast/

I am still on the fence as to if this screencast stuff is actually useful. Maybe it's all just web-2.0 style style-over-substance crap. Comments appreciated and will help me figure out how much effort to put into video content vs. straight up blog or technical writing.

Open Source Grid & Cluster Conference Photostream

dag@sonsorol.org (chris) — Wed, 14 May 2008 14:17:30 -0400

I'll post links to talk slides shortly, meanwhile a photo stream from the event can be found here:
http://flickr.com/groups/opensourcegridcluster/