I have made a pull request in cascading for the changes I made to support Ambrose ... But I get no reply till now https://github.com/cwensel/cascading/pull/26

Hi,

I'd like add Hive runtime support for this great tool. Some notes about the implementation:

• The job progress notification is done by using Hive's (0.11.0+) Hook interfaces
• One Hive statement represents one workflow. All statements are executed in a script sequentially, those which result in a MR job are visualized. At the end all workflows can be replayed
• Currently Ambrose uses Hadoop 0.22. Note, that Hive is not compatible with this version, therefore I set Hadoop 0.23.x for Ambrose-hive (also tested with Hadoop 1.x.x)

More information can be found in hive/README.md. I hope you'll find this extension useful, any suggestions, remarks..etc are very welcome.

regards, Lorand

Hello Bill, Andy, Nicolas, Our work focused on extending cascading to allow it to be supported by Ambrose. The members of the team who added this extension are Ahmed Eshra and Ahmed Mohsen. We did this work as part of our project that we did for the class taken at Alexandria University and taught by Iman Elghandour: http://www.alexeng.edu.eg/~ielghand/teaching/cs713/. Hope that our code will be qualified to be added as part of Ambrose:) We are also working on some extensions to the UI with other classmates.

Sorry for the delay, as we worked on some issues in our code .

Added a draggable, resizable div to display the script for pig. When you mouse over the node or the row in the table, it auto scrolls to the section and highlights the aliases for you. Blue for mapper, yellow for reducer and green for combiner. The div is closable, and you can clear the color with the refresh button.

Another take on https://github.com/twitter/ambrose/pull/117, with lot of improvements

• getClusters in dashboard page based on hRavenClusters.properties
• get appId consistent with Hraven (using hraven's apis)

This code still suffers with jackson deserialization problem. That will be solved independently of this PR.

Most significant changes:

• All client javascript code has been refactored / rewritten
• All Azkaban graph layout java code has been removed
• The existing DAG view has been replaced

I've tested these updates extensively with the demo data, but only with a few actual scripts via pig local mode. However, stability should already be improved as the Azkaban code would through NPEs for some job graphs (those with disconnected components).

Still TBD:

• The chord and graph views don't resize with changes to viewport dimensions; The page must be reloaded to properly resize those views.
• The view colors and other theme related params should be centralized within core Ambrose object.

I've updated a bunch of 3rdparty lib versions, and attempted to fix the pig module, given pig internal api changes.

@billonahill, @julienledem if you have any feedback on the pig updates, I'd love to hear them.

Addresses #82 by adding a line of text between the "Workflows" h2 and the table of workflows itself which conforms to pattern:

Workflows on $cluster with status $status and user $user.

$status and $user default to "any".

There may be more elegant ways to work this into the design, but the dropdown menus within the navbar don't look at all good if I include their current value in the text label and dynamically change it when filter value is updated. The resulting shifting of other navbar elements is not so nice.

The elapsed time feature didn't work for Hive runtime. AmbroseHiveStatPublisher now instantiates MapReduceJobState with args where the elapsed time is initialized/maintained. Further fixes/improvements:

• counterGroupMap was missing from event file
• new demo example (has more pretty DAG)
• resolve internal join alias name
• HiveJobTest: add testing metrics in event json
• minor indentation issues

Also:

• Replaces ./bin/ambrose-package with mvn package
• Adds Eclipse Jetty to ambrose-pig deps and removes references to Hadoop installation materials from pig-ambrose invocation script
• Adds demo pig script and data to amrbose-pig
• Updates ambrose-demo script to work with new client web resources location
• Updates README.md quick start text

Currently, a cascading driver program that is integrated with an ambrose listener never actually terminates. The only way to shutdown an HRaveStatsWriteService is via a shutdown hook, eg

    Runtime.getRuntime().addShutdownHook(new Thread() {
      @Override
      public void run() {
        shutdown();
      }
    });

This works in pig, which explicitly calls System.exit(run(...)) where run drives the entire pig program. Scalding does not have such an explicit System.exit, so the executor pool hangs forever waiting for instructions.

This PR adds a StatsWriteService#shutdownWriteService method to gracefully shutdown the write service. Add a call to this method in the cascading onComplete method, which is called when all work for the flow is finished.