gridengine.info

Thanks to Jana, there is a new wiki page up at http://wiki.gridengine.info/wiki/index.php/ARCoQueries that documents some additional contributed ARCo queries for the Grid Engine Analysis & Reporting Console.

Use ARCo? SGE developers are seeking feedback on potential methods for opening up the database backend to accept data from multiple Grid Engine installations.

Read the initial report here:
http://gridengine.sunsource.net/files/documents/7/157/ARCo_Multiclustering.pdf

Roland writes:

I've added a new howto with a spreadsheet document to calculate the estimated database space usage. The link is:

http://gridengine.sunsource.net/howto/arco/arco_db_size.html

I appreciate your Feedback, especially about discrepancies with calculated and real world values, to improve the document.

The HowTo document contains nice spreadsheet where one can plug in values and see what the estimated size requirements may be.

For those that don't have OpenOffice installed or handy, A version converted to MS Excel 97 can be found here: http://gridengine.info/files/arco_db_size_v1.1.xls

There have been lots of interesting Wiki work and CVS commits over the past week or so, some highlights:

DRMAA 1.0 Java Bindings

Grid Engine CVS for the 6.1 release has been updated with Java bindings for DRMAA 1.0 support. People interested in Java language bindings for DRMAA have a very short time to review the 1.0 specifications and provide feedback. (via DanT's blog)

{draft} ARCo Howto's for MySQL and Oracle 10g committed to CVS

No formal announcement has been made regarding these documents so beware that they are drafts subject to change. The documents likely refer to versions of the Analysis and Reporting Console (ARCo) that will only be available in the upcoming SGE 6.1 snapshot and beta releases. That said though ...

• ARCo and MySQL Howto
• ARCo and Oracle 10g Howto | (CVS Head)

Boolean Expression Support - Implementation Details

A very detailed wiki page has been created to document the specifics of the new more flexible host and string resource expressions that will be supported in both the next point release of Grid Engine (6.0u10) and the upcoming 6.1snapshot2 releases:

http://wiki.gridengine.info/wiki/index.php/EvaluationExpressionSupport

Andy's email has not made it into the list archives yet so this post will quote directly from it ...

Executive summary:

• In the 6.0 branch, SGE release version 6.0u10 will be released in 2 weeks with more than 50 bug fixes and enhancements
• Grid Engine 6.1 (a significant update) will issue it's second preview/comment release within 2 weeks
• Grid Engine 6.1 will have a beta release targeted for the last week of February
• Grid Engine 6.1 is targeted for an official release in April 2007

From the email:

In addition to the new functionality already announced at

   http://gridengine.sunsource.net/news/SGE61preview-announce.html

this version will be feature complete. The following features will be added
with 6.1preview2:

   - support for boolean expressions for host and string resources:
        Example:
         % qsub -l arch="!sol-sparc64&(sol*|hp*) ..
         -> run job on architecures starting with "sol" or "hp"
         but not on architecture "sol-sparc64"

         The "usual" boolean support syntax with logical OR ("|"), AND
         ("&") and NOT ("!") and brace ("(", ")") will be supported.

         The characters  [, ], ?, *, &, | and ! will not anymore be allowed as
         values for strings

   - new "qsub -wd " feature
        an enhancement to "qsub -cwd" but the start directory of the job is
        given after the "-wd" option

   - MySQL support for ARCo
        We need the active testing and feedback from our community users
        to be able to decide of MySQL suport can become a fully supported
     part of the SGE 6.1 release.

   - FreeBSD support
        it's also part (at source code level) of the SGE 6.1 release. This
        contribution comes from our open source community.

        We welcome our community members to add courtesy binary builds to
        our open source project download page)

    - various nice qmon improvments from Hin-Tak Leung

         http://gridengine.info/articles/2007/01/12/meet-and-thank-hin-tak-leung

      These improvments also will be part of 6.0u10

Personally, I'll be spending time testing MySQL ARCO interoperability and learning more about the fabulous new Resource Quota functionality coming in SGE 6.1.

Sun has formally announced the additions promised at SC'06, the full announcement is available online here:

• http://gridengine.sunsource.net/news/Hedeby_ARCo_announce.html

Of the two, ARCo is the more established layered product. This is the SQL driven accounting and reporting tool that was previously only available in the commercial version of N1 Grid Engine from Sun. ARCo uses Java to parse the SGE accounting logs for inclusion into an SQL back-end database. In addition to the metrics found in the accounting logs, ARCo has hooks for calculating useful "derived" metrics that are not explicitly stored in the accounting files.

When I first used ARCo (early on in its very first release version) one of the main weaknesses was the front end web based reporting console - for anything but the most basic reports, a user was expected to paste raw SQL queries into a web form. Sun's act of putting ARCo into the open source codebase should hopefully kickstart an idea that has been floating around for a while -- some sort of community wiki page or repository of user-generated ARCo queries and report templates. ARCo users are encouraged to send these sorts of tips and tricks to the users mailing list.

"Project Hedeby" aka the "Grid Engine Service Domain Management module" also mentioned at SC'06 is at an earlier stage in it's development. The nontechnical description is as follows:

Project Hedeby provides access to a new technology which allows to dynamically manage resources across so called Service Domains. Service Domains can be envisioned as autonomous Grids controlled by a resource manager including but not limited to Grid Engine. Hedeby will adjust the allocation of resources to individual service domains in order to meet Service Level Objectives. Reallocating a host resource to another service domain may include re-provisioning of the underlying virtual or actual operating system stack.

In his interview with GRIDtoday, Fritz provides the following description:

"... provides policy and demand-based re-allocation of arbitrary resources across service domains. Service domains are totally autonomous Grids which are controlled by a workload management facility, such as Grid Engine, but also by arbitrary other service infrastructures like application servers or web servers..."

Thanks to Andy for pointing out that the project codename, "Hedeby" refers to a Viking trade town from the 8th-11th century.