gridengine.info

qstat full view (a)

qstat full view (b)

qstat queue summary

qstat resource summary

qstat view

The first thing you need to do is make sure you have the Java^TM platform, the latest JWSDP, Ant, and at least Grid Engine 6.0u7 (or an equivalent maintrunk source build) installed.

I am going to refer to several directories in this tutorial. They are:

Note that you don’t need to have the above set as environment variables in your shell, but if you do, you can just copy and paste commands from this tutorial.

The first step is to create an external bindings file. This file will provide the JAXB class generator with some additional information. Specifically, the external bindings file will 1) assign a package to the generated files and 2) fix a naming conflict in the qstat.xsd schema. Create a file called $SCHEMA_HOME/qstat.xjb with the following contents:

<?xml version="1.0" encoding="UTF-8"?>

<jxb:bindings version="1.0"
  xmlns:jxb="http://java.sun.com/xml/ns/jaxb"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">
   <jxb:bindings schemaLocation="qstat.xsd" node="/xs:schema">
      <jxb:bindings node="//xs:complexType[@name='job_list_t']">
         <jxb:bindings node="//xs:attribute[@name='state']">
            <jxb:property name="stateAttribute"/>
         </jxb:bindings>
      </jxb:bindings>
      <jxb:schemaBindings>
         <jxb:package name="com.sun.grid.xml.qstat"/>
      </jxb:schemaBindings>
   </jxb:bindings>
</jxb:bindings>

This file sets the package to com.sun.grid.xml.qstat, but you can set the package to whatever suits you. Keep in mind, though, that you may want to generate a set of bindings for each of the three schemas, and you don’t want them to overlap.

Next, you’ll generate the binding classes. You do that by running:

% $JAXB_HOME/bin/xjc.sh -d $BIND_HOME -b $SCHEMA_HOME/qstat.xjb $SCHEMA_HOME/qstat.xsd

If all went well, you’ll see a list of the classes that are generated. Congratulations! You now have a qstat XML binding!

What do you do with it? Well, let me get you started. First, for convenience, let’s create an Ant build script to compile the binding and run a sample app. Create a file called $BIND_HOME/build.xml with the following contents:

<?xml version="1.0" standalone="yes"?>
<project basedir="." default="compile">
  <path id="classpath">
    

  <!--compile Java source files-->
  <target name="compile" description="Compile all Java source files">
    <echo message="Compiling the java source files..." />
    <javac destdir="." debug="on">
      <src path="." />
      <classpath refid="classpath" />
    </javac>
  </target>

  <target name="run" depends="compile" description="Run the sample app">
    <echo message="Running the sample application..." />
    <java classname="Main" fork="true">
      <classpath refid="classpath" />
    </java>
  </target>
</project>

This build script is very primative. With any amount of Ant skills, you should be able to write something that better suits your needs. My goal here is only to cover the bare necessities.

Now, let’s create the sample app. Let’s write a simple app that lists the job number of name of all jobs currently in the system, called $BIND_HOME/Main.java. It might look something like this:

import java.util.*;
import com.sun.grid.*;
import javax.xml.bind.*;

public class Main {
    
   public static void main (String[] args) throws Exception {
      // Create a JAXB context
      JAXBContext jc = JAXBContext.newInstance ("com.sun.grid.xml.qstat");
            
      // Use the context to create an Unmarshaller
      Unmarshaller u = jc.createUnmarshaller();
            
      // Fork a qstat -xml
      Process p = Runtime.getRuntime ().exec ("qstat -xml");

      // Let the binding do it's magic
      JobInfo ji = (JobInfo)u.unmarshal (p.getInputStream ());
      List list = ((JobInfoT)ji.getJobInfo ().get (0)).getJobList ();
      Iterator i = list.iterator ();

      while (i.hasNext ()) {
         JobListT jlt = (JobListT)i.next ();

         System.out.println (jlt.getJBJobNumber () + ": " + jlt.getJBName ());
      }
   }
}

In the call to JAXBContext.netInstance(), we specify the package we used to generate the binding. What we get back from the call to Unmarshaller.unmarshal() is an object tree derived from the qstat output. We then walk the object tree until we get to the job list, and then we go through the jobs, printing the number and name for each one.

Clearly, the only way you will know what the object tree looks like is to 1) read the schema, 2) read the generated source files, or 3) generate JavaDocs from the generated source files. 3 is the best option, but 2 is what I actually did.

To build and run this sample app, do the following:

% ant -Djwsdp.home=$JWSDP_HOME build
% ant -Djwsdp.home=$JWSDP_HOME run

If you want to parse the output from qstat -j, you will need to repeat this process with the $SCHEMA_HOME/detailed_job_info.xsd schema. When processing this schema you will need to change the external bindings file a little. You’ll want a file called $SCHEMA_HOME/detailed_job_info.xjb that contains:

<?xml version="1.0" encoding="UTF-8"?>

<jxb:bindings version="1.0"
  xmlns:jxb="http://java.sun.com/xml/ns/jaxb"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">
   <jxb:bindings schemaLocation="detailed_job_info.xsd" node="/xs:schema">
      <jxb:schemaBindings>
         <jxb:package name="com.sun.grid.xml.job"/>
      </jxb:schemaBindings>
   </jxb:bindings>
</jxb:bindings>

Again, feel free to change the package. You’d use this binding the same way you use the other one. There is a third schema in the $SCHEMA_HOME directory, message.xsd, that you won’t likely need, but if you do, you can generate a binding for it just like you did for detailed_job_info.xsd

Clearly, what I have provided here is only a starting point. For more information about using JAXB, see the docs included with the JWSDP download. Particularly useful are the examples. An obvious next step would be to extend the build script to generate JavaDocs and to split the class files out of the source tree. Another good next step would be to customize the binding classes so that, for example, the status code gets returned as a list of strings instead of a binary or’ed int. If you have problems, let me know. This tutorial is still a work in progress, so feedback is welcome.

The bitmasks used for JAT_state are:

   JHELD                   0x00000010
   JQUEUED                 0x00000040
   JWAITING                0x00000800
   JRUNNING                0x00000080
   JSUSPENDED              0x00000100
   JSUSPENDED_ON_THRESHOLD 0x00010000
   JERROR                  0x00008000

Translated into decimal form (which is what XML qstat output contains) the values are:

  JHELD:                   16
  JQUEUED:                 64
  JWAITING:                2048
  JRUNNING:                128
  JSUSPENDED:              256
  JSUSPENDED_ON_THRESHOLD: 65536
  JERROR:                  32768

So, when qstat XML produces JAT_state=128 we know that this means the job is running (state "r" in the human readable qstat output). We also know that the bitmasks are ADDED to account for multiple applicable states in an efficient manner. This means that the user reported value of "JAT_state=2112" can be broken down into JQUEUED+JWAITING because 2048+128=2112.

The states "queued + waiting" translate into the familiar "qw" state that is known to all Grid Engine users who use qstat on the command-line.

Commentary: This frightens me because I am lazy and not a good software engineer. heh. I understand how useful bitmasks are for software, the sum of any bitmask value will be unique which allows Grid Engine to rapidly and efficiently store and compute upon various status and states. The problem for me comes down to this: When faced with JAT_state=(some integer) how do I decompose that integer back into useful human-readable information about the relevant state or states? This is easy when a single bitmask is used but when the value is a SUM of a bunch of bitmasks it will be harder. I'll probably take the lazy way out and keep a lookup table of common sums (like 2112='qw'). Anyone have any better ideas? How would one handle this in the context of an XSL styleheet that is supposed to translate qstat XML into XHTML, text or PDF form?