gridengine.info :

Keeping single slot jobs off of certain nodes

dag@sonsorol.org (chris) — Thu, 08 May 2008 16:01:07 -0400

"I'm looking at finding a way to either limit single-slot jobs, or requiring all jobs in a given queue to be running in a pe. Specifically, I have some SMP nodes, that I'd rather not waste on single thread, and also keep the single thread jobs off of the infiniband connected nodes. I have gigE small cpu count nodes for this task."

Dan replied with another example of clever use of the new SGE Resource Quota syntax within SGE 6.1 and later:

You can use resource quota sets to restrict non-PE jobs to certain queues hosts.
limit pes !* hosts @smp to slots=0

Slick!

Think I'm going to like the new Sun wiki

dag@sonsorol.org (chris) — Thu, 08 May 2008 15:34:56 -0400

One of the more interesting things (to me at least!) in the recent news about the SGE 6.2 beta was the word that all documentation and manuals would be moving to a new home at http://wikis.sun.com.

I registered a user account a few days ago and hit the site today to see if any SGE stuff had made it over. The screenshot below is what I found. It's nice to see smart tech people have a sense of humor. The downtime must have been short as when I I refreshed the browser the site was back to normal.

SGE 6.2 goes beta next week (your help needed)

dag@sonsorol.org (chris) — Mon, 05 May 2008 10:00:43 -0400

SGE 6.2 is being released in Beta form next week and the developers are asking for people to make some time if possible to fully test out the beta snapshot of the latest major SGE point release.

Andy's full note can be found here (well worth reading in full ...):
http://gridengine.sunsource.net/servlets/ReadMsg?list=users&msgNo=24426

In my mind, I'm most excited about the following:

Advance Reservations & array job inter-dependencies
The scheduler is now a thread within the qmaster!
The JVM running within the qmaster
SGE moving all docs into wiki form!

SGE testbeds: Simulate mass numbers of exec hosts

dag@sonsorol.org (chris) — Fri, 02 May 2008 08:42:00 -0400

Interesting message on the developers list recently as a comment attached to Issue 2364. Within, Andreas explains the use of SIMULATE_EXECDS=true parameter that allows unrestricted execution host creation (via suppressing unknown host errors).

I can see this as being very useful for testing SGE scheduler and policy configuration settings before implementing them on production systems.

From the comment:

This is a short HOWTO for the use of the cluster simulator:

(1) Start with installing a new SGE cluster as used, but
install not more than the qmaster itself

(2) After successful installation use qconf -mconf to set

    SIMULATE_EXECDS=true

in qmaster_params section of sge_conf(5). This causes the
suppression of the 'unknown' queue states.

(3) Make sure the "all.q" and any other queue that you
configure does not use any 'load_threasholds'. Cluster
simulator has no means to anyhow emulate load values. As a
result there will be no load values. For that reason
load_threasholds may not be used as it would cause load
alarm queue states that prevent scheduler from dispatching
jobs into your queues.

(4) Use qconf -ae|-Ae to create arbitrary number of
simulated execution hosts. The hosts needs not exist as
qmaster anyways won't try to send anything to it, but the
hostname must be resolvable.

Optionally:

(5) If you care for scheduler runtimes set

     PROFILE=true

in the params section of sched_conf(5) using qconf -msconf.

Now your simulated cluster is ready. You can send in
arbitrary numbers of jobs. Due to (2) and (3) scheduler will
dispatch them and send corresponding orders to qmaster.
Qmaster will behave as if it would start the jobs, but it
raise timers to ensure job state transitions are passed as
used. What won't work is interactive jobs (i.e. qrsh, qsh
etc.) and parallel jobs with control_slaves set to true in
sge_pe(5). Jobs' runtime can be controled via the first job
argument. That means when

# qsub -b y /bin/sleep 5

is submitted, the job will finish after five seconds.

RHEL5.2/Centos5 kernel update may cause problems

dag@sonsorol.org (chris) — Mon, 21 Apr 2008 12:20:25 -0400

This is a heads up for RedHat Enterprise Linux (RHEL) users as well as for users (like myself) of the various Centos variants.

There is a recent patch for RHEL that changes the inode data structure exposed to NFS clients from 32 bits to 64 bits in size. The basic summary of this issue is that many applications may not handle this change gracefully (such as one report with the SGE linux binaries.)

RHEL and modern Centos users should probably pay attention to (by subscribing as CC: contacts) to this issue:
http://gridengine.sunsource.net/issues/show_bug.cgi?id=2543

A RedHat bug report discussing the issue in more detail is here:
"Large inode number patch breaks applications"
https://bugzilla.redhat.com/show_bug.cgi?id=241348

mpiblast, SGE and MPICH2 integration

dag@sonsorol.org (chris) — Mon, 21 Apr 2008 11:49:28 -0400

Matthias Neder has posted a quick summary of a tightly integrated MPICH2 integration that can successfully handle his mpiblast application integration.

The summarized solution can be found here:
http://gridengine.sunsource.net/servlets/ReadMsg?listName=users&msgNo=24204

6.2 Bringing Significant Improvements to Cluster Queue Matching

dag@sonsorol.org (chris) — Wed, 16 Apr 2008 07:21:46 -0400

Interesting writeup from Andreas Hass reproduced in full below ...

I thought this could be of interest for those who care for dispatching times. This maintrunk check-in

http://gridengine.sunsource.net/servlets/ReadMsg?list=cvs&msgNo=9814

will improve the matching times for set-ups where queue resource limits such as -l h_rt or -l h_vmem are criterion whether a job gets into a queue or not.

Before the above change we had an exponential growth of dispatching times

04/15/2008 11:48:01|schedu|es-ergb01-01|P|PROF: job dispatching took 0.030 s (20 fast, 0 comp, 0 pe, 0 res)
04/15/2008 11:48:23|schedu|es-ergb01-01|P|PROF: job dispatching took 0.130 s (40 fast, 0 comp, 0 pe, 0 res)
04/15/2008 11:48:50|schedu|es-ergb01-01|P|PROF: job dispatching took 0.630 s (80 fast, 0 comp, 0 pe, 0 res)
04/15/2008 11:49:26|schedu|es-ergb01-01|P|PROF: job dispatching took 3.210 s (160 fast, 0 comp, 0 pe, 0 res)

now growth is linear

04/15/2008 11:53:54|schedu|es-ergb01-01|P|PROF: job dispatching took 0.000 s (20 fast, 0 comp, 0 pe, 0 res)
04/15/2008 11:54:17|schedu|es-ergb01-01|P|PROF: job dispatching took 0.020 s (40 fast, 0 comp, 0 pe, 0 res)
04/15/2008 11:54:44|schedu|es-ergb01-01|P|PROF: job dispatching took 0.050 s (80 fast, 0 comp, 0 pe, 0 res)
04/15/2008 11:55:16|schedu|es-ergb01-01|P|PROF: job dispatching took 0.070 s (160 fast, 0 comp, 0 pe, 0 res)

also note this maintrunk check-in

http://gridengine.sunsource.net/servlets/ReadMsg?list=cvs&msgNo=9713

when profiling is enabled in sched_conf(5) like this

   :
  params  PROFILE=true
   :

the actual cause for exponential/linear dispatching times becomes fairly obvious: Without the above improvement the scheduler did check each single queue instance also in cases when the entire cluster queue was not suited

04/15/2008 11:48:01|schedu|es-ergb01-01|P|PROF: sequential matching global          rqs     cqstatic      hstatic      qstatic     hdynamic qdyn
04/15/2008 11:48:01|schedu|es-ergb01-01|P|PROF: sequential matching 20            0           30          200          210          210 65
04/15/2008 11:48:23|schedu|es-ergb01-01|P|PROF: sequential matching 40            0           60          800          820          820 230
04/15/2008 11:48:50|schedu|es-ergb01-01|P|PROF: sequential matching 80            0          120         3200         3240         3240 860
04/15/2008 11:49:26|schedu|es-ergb01-01|P|PROF: sequential matching 160            0          240        12800        12880        12880 3320

now the queue instances checked is done only if needed

04/15/2008 11:53:54|schedu|es-ergb01-01|P|PROF: sequential matching global          rqs     cqstatic      hstatic      qstatic     hdynamic qdyn
04/15/2008 11:53:54|schedu|es-ergb01-01|P|PROF: sequential matching 20            0           30           10           10           10 10
04/15/2008 11:54:17|schedu|es-ergb01-01|P|PROF: sequential matching 40            0           60           20           20           20 20
04/15/2008 11:54:44|schedu|es-ergb01-01|P|PROF: sequential matching 80            0          120           40           40           40 40
04/15/2008 11:55:16|schedu|es-ergb01-01|P|PROF: sequential matching 160            0          240           80           80           80 80

the setup I used to get these numbers was two cluster queues over 10,20,40,80 simulated hosts (using SIMULATE_EXECDS=true in sge_sconf(5) qmaster_params).

One cluster queue was not eligible at all for any of the jobs due to a queue resource limit, but I forced scheduler to evaluate this cluster queue first with sequence numbers as to provoke the worst case. In addition each job requested slightly different resource amount

  -l h_rt=300
  -l h_rt=301
  -l h_rt=302
     :
  -l h_rt=379

as to sabotage the reuse of dispatching results for identical jobs within a scheduling interval.

Regards, Andreas

6.1 leak found; schedd_job_info is not your friend

dag@sonsorol.org (chris) — Thu, 10 Apr 2008 11:07:00 -0400

Anyone interested in the memory leak that has been bothering some 6.1 users should check out the comments associated with Issue #2464:
http://gridengine.sunsource.net/issues/show_bug.cgi?id=2464

Among the interesting things you'll see are:

A great example of motivated SGE users and developers working together to track down a hard to find problem
Interesting comments on the potential "unfixible" (my words) nature of the schedd_job_info messages
A really cool workaround for getting job scheduler messages with schedd_job_info=FALSE

In a nutshell, there is a problem in the schedd_job_info framework that can cause massive resource utilization on the qmaster machine. This happens in particular on larger systems or places with large numbers of queue instances. This can also pop up on systems with jobs that are pending due to un-fulfillable resource requests. This explains why I saw the memory leak on my small testbed cluster -- I have a number of "pend forever" jobs in the queue for demonstration purposes.

The fix is to disable schedd_job_info. This is potentially problematic though as that feature is pretty much my goto-first action for troubleshooting job dispatch problems.

However, in a recent update comment to this issue, andreas added a possible tip for getting scheduling messages about a job in a way that that puts far less load on the system AND does not require schedd_job_info=TRUE:

qalter -w v

Remember though that comments found in a bug report are not "gospel" so don't read this as news that schedd_job_info is forever broken or going away. Expect to see this and other issues discussed as part of the SGE Roadmap. You are attending the May 2008 SGE Workshop, right?

Release 6.1u4 is out

dag@sonsorol.org (chris) — Fri, 04 Apr 2008 10:06:11 -0400

Congratulations to the SGE developer team!

Big news today -- 6.1u4 was just announced; hopefully addressing some persistent issues people have been having with the previous releases. The plaintext list of fixed issues can be found here:
http://gridengine.sunsource.net/project/gridengine/61patches.txt

The full announcement is here:
http://gridengine.sunsource.net/news/GE61u4-announce.html

I've been unable to keep 6.1u3 running consistently on a small test system, probably due to the same memory leak others have been reporting. There is a chance that a subtle leak still exists or at least has not been fully tracked down in 6.1u4 but multiple people are working diligently on this. Best bet is to monitor the users mailing list to see the feedback.

Summer 2008 SGE Training Workshops

dag@sonsorol.org (chris) — Sat, 22 Mar 2008 16:27:00 -0400

Hi folks. It's done. I've made a personal and financial commitment to organize a regularly occurring series of Grid Engine Training Workshops starting initially in the Cambridge, Massachusetts area. This is a darwinian test to see if my thoughts about the size and needs of the Grid Engine community are true. With existing SGE training opportunities only scheduled 1-2 times per year it is still an open guess as to the size of the potential audience for these sorts of events.

Course details and updates will always be here:
http://blog.bioteam.net/category/training/

Download the brochure here:

Obviously my employer has a bit of a commercial/profit motive here but I've been the one pushing to make this happen. Consider this a bit of market research to see if BioTeam should invest in growing the number of staff capable of providing SGE related training, professional services and support to the community at large.

I welcome your comments and feedback and would appreciate any assistance in spreading the word about these events.

Thanks! -- Chris.

CFP: Open Source Grid & Cluster Conference 2008

dag@sonsorol.org (chris) — Wed, 19 Mar 2008 10:15:28 -0400

Reminder: Call for Participation closes Friday, March 21

OPEN SOURCE GRID & CLUSTER CONFERENCE 2008

Featuring: GlobusWorld, Grid Engine Workshop, Rocks Cluster Workshop

May 13 - 15, 2008 in Oakland, California
http://www.OpenSourceGridCluster.org

DEADLINE FOR ABSTRACT SUBMISSIONS: March 21, 2008

Whether you are a Grid or Cluster expert with technical advice to share, or a leader with visions for the future of open source Grid and Cluster computing in research or industry, the Open Source Grid & Cluster Conference is the premier event for delivering your message to the Grid and Cluster community. In past years, hundreds of Grid and Cluster professionals from research and industry have attended individual events such as GlobusWorld, the Grid Engine Workshop, and Rocks-a-Palooza to discuss Grid and Cluster adoption issues, to receive training and exchange information related to these widely used Grid and Cluster software systems. This year the Globus, Grid Engine, and Rocks communities are joining forces to create the most comprehensive event on open source Grid and Cluster computing to date.

The Open Source Grid & Cluster Conference program will offer a wide variety of conference sessions, mini-symposiums, panel discussions, workshops, and tutorials. Speaking opportunities range from highly technical research, development, and deployment presentations to targeted panels on commercial and research adoption considerations. The Open Source Grid & Cluster Conference will run parallel tracks, some focused on Globus, Grid Engine, and Rocks community-specific topics, and others focused on cross-cutting and other open source Grid and Cluster software technologies and uses.

KEY DATES AND DEADLINES
Abstract submission deadline - March 21, 2008
Acceptance notification - April 15, 2008
Presentation Slides Due - April 30, 2008
SPEAKING TOPICS
Submissions should be centered on the theme of uses and implementation of Open Source Software for Grid and Cluster Computing.

All proposals should be submitted online at http://www.OpenSourceGridCluster.org/CFP.html

Click on through for the submission guidelines ...

Questions should be sent to program@OpenSourceGridCluster.org

SUBMISSION GUIDELINES ---------------------

ABSTRACT GUIDELINES
All submissions must include an abstract of no more than 500 words, and a brief bio for each presenter. Abstracts should be written so as to be self-contained and to provide the technical substance required for the program committee to evaluate the session's contribution to the Open Source Grid and Cluster community. Please indicate whether the proposed session is specific to just one of Globus, Grid Engine, or Rocks. If the presentation was given at another conference, then the name, date, and location of the event must be noted in the submission. Abstracts should be submitted in plain text format either as an attachment or in the main body of the e-mail. Abstracts and bios for accepted submissions will be published on the Open Source Grid & Cluster Conference website and in other conference material as the description of the session. Presentation slides may be published on the Conference website and distributed with conference material.

PRESENTATIONS
Presentation proposals may be submitted for individual time slots of thirty minutes. Please be sure to allow ten minutes for Q&A within this allotted time. Individual presentations will be grouped with similar topic presentations to fill an entire session.

BUILD YOUR OWN SESSION
Participants are invited to organize their own, complete, ninety-minute session, including but not limited to the following categories. The submission must include an agenda, and the names and associations of all participants.

Panel Session / Mini-Symposium: These sessions will enable conference attendees to learn from a group of experts on a particular topic. The session organizer may deliver an opening talk to set the context for the remainder of the session. Panelists will then give presentations designed to stimulate audience participation, on their preferably diverse opinions, experiences or expertise regarding the theme of the session. At least ten minutes should be reserved at the end for questions from the audience.

Birds-of-a-Feather (BOF) Sessions: These sessions will allow conference attendees to discuss focused subject areas. The session may include presentations and open discussion. Session organizers will be responsible for moderating these sessions and reporting on their outcomes.

WORKSHOPS AND TUTORIALS
Ample room is available for half-day and full-day pre-conference (Monday) and post-conference (Friday) workshops and tutorials. Workshops may include topical meetings with open registration or community/group meetings with resricted attendance. Tutorials may be on any topic related to the Open Source Grid and Cluster theme of the conference. Submissions must include preferred and minimum acceptable room size, and preferred and acceptable times. An extra nominal fee may be required of attendees or the organizer to cover additional costs such as A/V and food.

All proposals should be submitted online at http://www.OpenSourceGridCluster.org/CFP.html

Questions should be sent to program@OpenSourceGridCluster.org

Clever job prioritization tip

dag@sonsorol.org (chris) — Thu, 13 Mar 2008 13:28:48 -0400

Grid Engine has a built-in priority mechanism that is useful for allowing end users to sort and prioritize their own personal pending tasks -- this gives the users the ability to submit many jobs but still dictate which of those jobs need to be run more urgently than the rest.

In practice, though, this is actually fairly clunky to implement. By default the following conditions exist:

SGE will accept a priority range of -1023 to 1024
By default all jobs get assigned a value of 0
Only SGE managers can assign priority values higher than 0
Normal users can only assign negative priority values

See where we are going here? By default, a non privileged user can only describe some of her jobs as "less important" than others. There is no mechanism (besides granting the user SGE manager authority) for her to say "this job of mine is more important than that other pending job of mine...".

This is, ummmm, awkward to say the least and works in a way that is 100% opposite from what a sensible user or SGE Admin would expect. Users can only decrease the relative priority of their job in the default environment.

A recent mailing list post from Jeff highlights a nice little workaround. Jeff describes creating an entry in the sge_request file that automatically assigns a value of -p -100 to all submitted jobs that don't override the default with their own use of the -p switch.

This is a nice approach because by default it harms nobody (as all jobs have -p -100. Yet it gives headroom for a non privileged user to use the priority range -99 to 0 to designate some of her jobs as more personally important than others.

Background reference: manpage for sge_request.

Clever sharetree usage for project-based job priority grouping

dag@sonsorol.org (chris) — Wed, 12 Mar 2008 13:23:41 -0400

Recently on the mailing list there has been a discussion centering on how to order jobs within a project. For some reason, Daire's reply did not make it into the previous thread list. It is an interesting approach. Daire's method allows groups of jobs within a project to have different levels of priority entitlements in a way that does not interfere with other shares or projects. It also allows users or project leaders to more easily reallocate priorities on the fly, simply by changing the project associated with a task.

Daire writes:

...we decided to abstract gridengine's projects into priority groups
so that you can order jobs within a project by changing the job's project
(qalter -P). So for example if you have 2 projects A and B your 
sharetree might look something like:

ROOT
|-- A (75)
|   |-- A_1 (P) (10)
|   |-- A_2 (P) (1000)
|   |-- A_3 (P) (100000)
|   |-- A_4 (P) (10000000)
|   `-- A_5 (P) (1000000000)
`-- B (25)
   |-- B_1 (P) (10)
   |-- B_2 (P) (1000)
   |-- B_3 (P) (100000)
   |-- B_4 (P) (10000000)
   `-- B_5 (P) (1000000000)

where (P) signifies a gridengine "project" and the numbers in ()'s
are the assigned share values. Now you can move a job within project 
A between the 5 priority levels without effecting project B's share. 
Maybe use the Functional policy to ensure equal shares between users 
in the same priority band? Not sure how much the halflife factor will 
mess with the priority bands but the I'm assuming the large share 
differences between them will override the effect...

Similar discussions occur on this mailing list thread.

Reducing scheduler memory usage with libhoard

dag@sonsorol.org (chris) — Thu, 06 Mar 2008 09:45:00 -0500

It’s pretty interesting subscribing to the SGE Issues mailing list. This comment on Issue 2464 came across the wire today:

… I installed libhoard.so (http://www.hoard.org/) and started sge_schedd with it (changing the sge_schedd starting line in sgemaster to "LD_PRELOAD=/opt/hoard-3.7.1/lib64/libhoard.so sge_schedd").

There seems to be some problems with malloc and threads not freeing memory (or something similar, Andreas could explain this the right way) which could be affecting sge_schedd.

Since restarting sge_schedd using hoard I didn’t have any memory problems anymore, but this just happened one day ago.

If anyone else tries this method I’d appreciate feedback and comments.

Screenshots of enhanced Olesen FLEXlm tools in action

dag@sonsorol.org (chris) — Thu, 06 Mar 2008 09:21:00 -0500

In a follow-up post to Mark's recent announcement we've gotten our hands on some screenshots from Mark showing his tools in use. The screenshots show the results of using XSLT transformations to turn Grid Engine XML data into XHTML form suitable for web pages. The benefit includes web-based visibility into current resource (and software license!) usage. This is exactly the approach that I tried out with the xml-qstat project. Mark is pretty familiar with that effort and will be merging his improvements and enhancements into xml-qstat's SVN repository. Speaking personally as a "scratch an itch" programmer with no real software engineering skill or talent I'm pretty excited to have a real coder take a look at xml-qstat. Related to that I already owe a debt to Petr Jung from Sun who contributed the Java based CommandGenerator code that finally allows xml-qstat to be a 100% Java/Cocoon web application that does not require external perl daemons to cache XML state data.

Before the screen captures, I'd like to ask a favor of people who read this blog. I filed bug Issue #2335 back in July of 2007 and it has not received much love (or even a targeted milestone date for a fix). The bug is a simple one -- "qstat -f -xml" no longer reports load average data which (a) makes xml-qstat a whole lot less useful and (b) breaks the SGE developer philosophy of ensuring that command output returns the same information regardless of output format. Until that bug is fixed it does make sense for xml-qstat to have it's long overdue "1.0" release. If you have a user account over on http://gridengine.sunsource.net I'd appreciate it if you can cast one of your "votes" for Issue 2335. Thanks!

And now the screenshots (edited to mask out personal/company information). Click on each image for a larger version.

qhost overview

Click on through for the rest of the pictures ...

qstat full view (a)

qstat full view (b)

qstat queue summary

qstat resource summary

qstat view