SGE utilities from Duke SCSC
https://wiki.duke.edu/display/SCSC/SGE+Tools
Hat tip: Ed L. from MRL Boston who first pointed me towards the 'jobpar' link on the web. Big thanks also to John Pormann from Duke who took the time to make these utilities available to the community under an MIT open source license.
Tracking & rollback of SGE config changes
Ed Dale has a great article showing how he uses a subversion ("SVN") repository in conjunction with the SGE-supplied 'save_sge_config.sh' script to provide versioning and rollback capabilities for a Grid Engine installation.
The full writeup is here, well worth a read.
http://scompt.com/blog/archives/2009/10/13/versioned-grid-engine-configuration
Ed's work is a perfect companion to this recent mailing list thread where we discussed the need for comments and log messages that accompany SGE queue instance disablement and other state changes. The end result of that is renewed focus on the following open SGE enhancement requests:
- http://gridengine.sunsource.net/issues/show_bug.cgi?id=1539
- http://gridengine.sunsource.net/issues/show_bug.cgi?id=2179
- http://gridengine.sunsource.net/issues/show_bug.cgi?id=3161
If you agree with the above RFE requests, please use your collabnet votes to express your opinions.
Key FlexLM license integration tools updated
Mark has updated his code for making Grid Engine aware of FlexLM license servers. Read the full announcement here:
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=37&dsMessageId=221361
Without a doubt this is currently the industry best practice way of dealing with SGE/FlexLM integration issues. Kudos to Mark O. for open-sourcing his work.
DRMAA-python module updated
Enrico Sirola reports that an updated python module for interacting with DRMAA-compliant distributed resource management ("DRM") systems has been released.
The DRMAA working group website is http://www.drmaa.org/ for those looking for additional information.
MarkMail: Mine the grid engine maillist archives
MarkMail has just imorted all of the Grid Engine mailing lists from http://gridengine.sunsource.net into their archive, search, index and database system. Initial results are pretty impressive based on a few minutes of searching and experimentation -- seems like a great way to search the mailing lists for answers and info.
Click on the image above and you'll be take to a search on the term 'rqs'. Leave a comment with your impressions if you are so inclined.
MarkMail: Mine the grid engine maillist archives
MarkMail has just imorted all of the Grid Engine mailing lists from http://gridengine.sunsource.net into their archive, search, index and database system. Initial results are pretty impressive based on a few minutes of searching and experimentation -- seems like a great way to search the mailing lists for answers and info.
Click on the image above and you'll be take to a search on the term 'rqs'. Leave a comment with your impressions if you are so inclined.
Screenshots of enhanced Olesen FLEXlm tools in action
In a follow-up post to Mark's recent announcement we've gotten our hands on some screenshots from Mark showing his tools in use. The screenshots show the results of using XSLT transformations to turn Grid Engine XML data into XHTML form suitable for web pages. The benefit includes web-based visibility into current resource (and software license!) usage. This is exactly the approach that I tried out with the xml-qstat project. Mark is pretty familiar with that effort and will be merging his improvements and enhancements into xml-qstat's SVN repository. Speaking personally as a "scratch an itch" programmer with no real software engineering skill or talent I'm pretty excited to have a real coder take a look at xml-qstat. Related to that I already owe a debt to Petr Jung from Sun who contributed the Java based CommandGenerator code that finally allows xml-qstat to be a 100% Java/Cocoon web application that does not require external perl daemons to cache XML state data.
Before the screen captures, I'd like to ask a favor of people who read this blog. I filed bug Issue #2335 back in July of 2007 and it has not received much love (or even a targeted milestone date for a fix). The bug is a simple one -- "qstat -f -xml" no longer reports load average data which (a) makes xml-qstat a whole lot less useful and (b) breaks the SGE developer philosophy of ensuring that command output returns the same information regardless of output format. Until that bug is fixed it does make sense for xml-qstat to have it's long overdue "1.0" release. If you have a user account over on http://gridengine.sunsource.net I'd appreciate it if you can cast one of your "votes" for Issue 2335. Thanks!
And now the screenshots (edited to mask out personal/company information). Click on each image for a larger version.
qhost overview
Click on through for the rest of the pictures ...
qstat full view (a)
qstat full view (b)
qstat queue summary
qstat resource summary
qstat view
Advanced Reservation plugin for Grid Engine
Yoshio Tanaka posts the following:
... We are pleased to announce that advance-reservation plugin module called PluS version 1.0.0 RC 1 is now available for download at the PluS home page at: http://www.g-lambda.net/plus/ . PluS (Plug-in Advance Reservation Manager for Torque and Grid Engine) adds an advance-reservation function to Torque and Grid Engine. For SGE, one of the following operations will be performed based on the startup option. (1) SGE queue base version - The SGE schedule is not replaced, and the reservation function is realized simply by managing the reservation queues. (2) SGE self scheduling version - The original SGE scheduler is replaced by the PluS SGE scheduler which realizes the reservation management function and the job scheduling function. ...
The package is released under the Apache 2 License. It appears that the system has mainly been developed and tested on the following configuration: Linux 2.6.x, Intel x86, glibc 2.3.3, SGE 6.0u8
The HTML version of the PluS Manual is online here:
http://www.g-lambda.net/plus/wp-content/uploads/2006/10/manual.html.
The http://www.g-lambda.net/plus/ site contains a link to a PDF from a IEEE conference paper covering the system in more technical detail.
OpenDSP announcement: Web services access to SGE via DRMAA
Interesting announcement on the mailing lists today regarding http://sourceforge.net/projects/opendsp.
From the website:
"OpenDSP is an open architecture implementation of SOAP Web Service multi-user access and policy-based job control using DRMAA (Distributed Resource Management Application API) routines supported by various distributed resource management systems."
For version v0.1 RC1 the feature list is pretty interesting. Hopefully I'll have time to experiment with this a bit.
XML Feeds