gridengine.info

%  jrunscript -cp $SGE_ROOT/lib/drmaa.jar -f drmaa.js

Job 2 submitted
Job 2 has ended
Job terminated abnormally
%

Richard Hierlmeier seems to have joined the ranks of Sun Bloggers and has a facinating post up documenting how he used the javascript engine that ships with Java 6 to bind to drmaa.jar.

The post is here:

http://blogs.sun.com/rhierlmeier/entry/java_drmaa_binding_with_javascript

Cheng Soon Ong mentions on the developer mailing list:

I've been playing around with python and sge, and translated the drmaa tutorial at http://gridengine.sunsource.net/project/gridengine/howto/drmaa.html to a python version. Look at sge_tutorial.rst in the attached archive.

The newly translated tutorial will likely find a home on gridengine.sunsource.net soon but in the mean time you can:

• Read the original mailing list post
• Read the follow-up post containing a patch
• Download the tutorial files

Most casual SGE users and admins probably find little cause to monitor the Grid Engine developer mailing list. A nice little success story has played out on the list recently with a user assisting the SGE dev team in quickly discovering, isolating and fixing a memory leak that has been in the codebase since the DRMAA 1.0 API release.

A user posted this message to the developer list, showing what appears to be a memory leak in in drmaa_run_job(). Andreas then replied asking if it was possible for the user to recreate the issue while running under the valgrind instrumentation framework.

In this follow-up thread, the user-provided valgrind data allowed Andreas to pinpoint the problem, file Issue #2497 with the bug tracking database and then post a preliminary patch that fixes the problem.

The patch still needs to undergo code review before it makes it officially into the Grid Engine codebase. Overall this is a nice little success story where a user was able to go the extra mile (by instrumenting under valgrind) in order to provide the developers exactly what they needed to quickly identify and fix things.

Kudos to James & Andreas.

The 22nd Open Grid Forum -- OGF22
Hyatt Regency Cambridge
Cambridge, MA USA
February 25-28, 2008
Website: http://www.ogf.org/OGF22/

Coworker Chris Dwan and I will be attending this event and one of us will likely end up speaking. Both of us are known to be somewhat cynical of the "big G" Grid Computing world so we'll be bringing our industry-centric views and bias towards practical solutions into the forum.

(I swear, when you speak to some of these "big G grid" supercomputing or academic folks you get the sense that they think that everyone has 100 million in government funding and a petabyte-scale single namespace storage solution to apply to the problems at hand....)

Among the other attendees that I know about, Chris Smith from Platform Computing will also be there -- he handles Platform's involvement with standards bodies and is another person on my "smart people that I learn a lot from" list. Should be an interesting event.

And finally, some discount registration offers for readers of this blog:

• "Buy one 1 pass and get a 2nd for free"
• "$150 discount off the purchase off the full day pass"

DanT has an interesting blog post up where he talks about two different methods getting down into invoking product-specific DRM features when using DRMAA clients. The article is a result of a SGE mailing list thread where someone found themselves wanting to invoke SGE-specific abilities that are not explicitly part of the DRMAA spec. Dan talks about the two main ways to go about doing this.

The full post is here:
http://blogs.sun.com/templedf/entry/using_drm_specific_functionality_via

On a related note, the "Writing Grid Enabled Applications" lab that Dan presented at JavaOne is now available online. You will have to register at the Sun Developer Network in order to view the video or download the materials. A nice walkthrough of DRMAA usage and the API.

There have been lots of interesting Wiki work and CVS commits over the past week or so, some highlights:

DRMAA 1.0 Java Bindings

Grid Engine CVS for the 6.1 release has been updated with Java bindings for DRMAA 1.0 support. People interested in Java language bindings for DRMAA have a very short time to review the 1.0 specifications and provide feedback. (via DanT's blog)

{draft} ARCo Howto's for MySQL and Oracle 10g committed to CVS

No formal announcement has been made regarding these documents so beware that they are drafts subject to change. The documents likely refer to versions of the Analysis and Reporting Console (ARCo) that will only be available in the upcoming SGE 6.1 snapshot and beta releases. That said though ...

• ARCo and MySQL Howto
• ARCo and Oracle 10g Howto | (CVS Head)

Boolean Expression Support - Implementation Details

A very detailed wiki page has been created to document the specifics of the new more flexible host and string resource expressions that will be supported in both the next point release of Grid Engine (6.0u10) and the upcoming 6.1snapshot2 releases:

http://wiki.gridengine.info/wiki/index.php/EvaluationExpressionSupport

Update: 13 June 2006
Andreas has started a formal Ruby-DRMAA project:

…the DL based Ruby wrapper for shared DRMAA C bindings now has it’s own home page and CVS repository under

http://drmaa4ruby.sunsource.net/

Andreas writes:

  http://gridengine.sunsource.net/files/documents/7/89/drmaa_ruby.tar.gz

my first version of a DRMAA binding for Ruby. Thanks to Ruby/DL it was
not necessary to use SWIG. Open issues are

  * assumes DRMAA 0.95 instead of DRMAA 1.0
  * memory leak due to JobTemplate destructor not yet functioning
  * a number of functions are not yet supported (drmaa_ps_job(),
    drmaa_control(), ...)

In related news, it appears that people are using and beginning to extend the Ruby-based SGE log-file analysis script mentioned previously on this site.

Interesting announcement on the mailing lists today regarding http://sourceforge.net/projects/opendsp.

From the website:

"OpenDSP is an open architecture implementation of SOAP Web Service multi-user access and policy-based job control using DRMAA (Distributed Resource Management Application API) routines supported by various distributed resource management systems."

For version v0.1 RC1 the feature list is pretty interesting. Hopefully I'll have time to experiment with this a bit.

A common feature request of DRMAA users is the ability to reopen an existing session -- something that the current version does not allow.

DanT has committed code offering session reopening features for the DRMAA Java and C bindings. It should be available with the release of Grid Engine 6.0u8.

The full details (including important info on versions and limitations) can be found here:

http://blogs.sun.com/roller/page/templedf?entry=good_drmaa_news

Fresh from his move back to the USA, Dan has posted a couple of Sun blog entries on DRMAA and Grid Engine. DRMAA is a GGF API specification for "the submission and control of jobs to one or more Distributed Resource Management (DRM) systems". It is currently well supported with Grid Engine 6 and it seems that folks are busy with getting other systems to support DRMAA 1.0

The first of two recent DRMAA posts is titled "Porting the DRMAA Java Language Binding":

Dan says:"There's a been quite a bit of talk on various aliases (and over private email) recently about what's required to port the Grid Engine DRMAA JavaTM language binding to another DRM. Since that is an interesting topic, I figured I'd assemble all of the answers here for easy reference...(more) "

The second entry is titled "Running Job Scripts With DRMAA", this topic has been popping up quite a bit on the Grid Engine lists recently (1, 2):

Dan says:DRMAA is intended as a general purpose API, which means it has to assume as little as possible about the jobs it runs. Grid Engine recognizes two broad classes of jobs: scripts and binaries. A script is a text file that is to be run by a shell and which may have embedded SGE options in it. (Lines starting with #$ are parsed by Grid Engine at job submission time for embedded options. See the man page for more info.) A binary is anything else. The user controls whether a job is treated as a binary or a script with the -b (for binary) qsub option. The Grid Engine default is to assume that all jobs are scripts.

DRMAA, however, makes a different assumption. The minimum assumption that DRMAA can make is that jobs are opaque and cannot be parsed, i.e. that all jobs are binary. This assumption is exactly the opposite of the one Grid Engine makes. Because jobs aren't assumed to be scripts, there are a few extra steps required to running scripts through DRMAA...(more)