GridKa School 2009 related events

Added by Michal Owsiak, last edited by Michal Owsiak on Sep 17, 2009  (view change)

Kepler workflows for HPC/GRID at GridKa School 2009

Within this tutorial we will present following technologies:

  • Kepler workflows
  • Java API 4 HPC/GRID within Kepler
  • Migrating Desktop
  • MPI and interactive jobs

Detailed agenda for the meeting can be found at following location: GridKa 2009 - agenda for Java API 4 HPC-GRID tutorial.

1. Requirements for the tutorial

In order to take part within tutorial you need following:

  1. Java 1.6.0_16 environment. Installation related details can be found here: Installing Java Environment
  2. Kepler 1.0 application. Installation related details can be found here: Installing Kepler Application
  3. Java API 4 HPC/GRID libraries. Installation related details can be found here: Installation of HPC-Grid Related Actors Within Kepler - GridKa2009 tutorial
    IMPORTANT NOTICE - jar conflicts

    Please, pay attention to the conflicting JARs section while installing Java API 4 HPC/GRID

  4. Demo workflows, that are available at following location: Kepler Workflows
  5. You can always refer to the demo movies that are available at following location: Kepler Movies

2. Executing simple workflows

In order to execute workflow, you have to load workflow XML file into Kepler. During this tutorial session we will use following workflows:

  • "Hello world" (you can download it from here link - it is also installed as: $KEPLER_HOME/demos/japi4hpcgrid/simple/Hello_World.xml)
  • "If-else" (you can download it from here link - it is also installed as: $KEPLER_HOME/demos/japi4hpcgrid/simple/if_else.xml)
  • "Loop" (you can download it from here link - it is also installed as: $KEPLER_HOME/demos/japi4hpcgrid/simple/array_loop_composite_elements.xml)

2.1 Hello world workflow

2.1.1 Using already existing workflow
Exercise no. 1 (approx. 5 min)

In this exercise you will execute simple Kepler workflow. In order to this follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. open "Hello world" workflow by issuing: File -> Open and navigate to:

$KEPLER_HOME/demos/japi4hpcgrid/simple/Hello_World.xml

3. after workflow is opened, press "Play" button

workflow should generate output within Display actor

If you are not sure how to proceed, you can watch demo movie at following location: movie

2.1.2 Building from the scratch
Exercise no. 2 (approx. 15)

In this exercise you will build simple "Hello World" workflow and execute it. In order to get this task done, follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. Type in "SDF" into "Search" field and press "Search" button

3. Drag and Drop SDF director into workflow

4. Right-click on SDF director and choose "Configure Director"

5. Set number of iterations to "1"

6. Type in "String" into "Search" field and press "Search" button

7. Add "String Constant" actor to the workflow

8. Right-click "String Constant" actor and choose "Configure Actor"

9. Type "Hello world!" into "value" field

10. Commit changes

11. Type in "Display" into "Search" field and press "Search" button

12. Add "Display" actor into workflow

13. Connect "String Constant" actor with "Display" actor

14. after workflow is ready, press "Play" button

workflow should generate output within Display actor

If you are not sure how to proceed, you can watch demo movie at following location: movie

2.2 If-else workflow

2.2.1 Using already existing workflow
Exercise no. 3 (approx. 10 minutes)

In this exercise you will execute simple Kepler workflow. In order to this follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. open "If-else" workflow by issuing: File -> Open and navigate to:

$KEPLER_HOME/demos/japi4hpcgrid/simple/if_else.xml

3. after workflow is opened, press "Play" button

workflow should generate output within Display actor

If you are not sure how to proceed, you can watch demo movie at following location: movie

2.2.2 Building from the scratch
Exercise no. 4 (approx. 20 minutes)

In this exercise you will build "if-else" workflow.

You should complete previous examples before starting this one

In this example it is assumed that you already know how to use actor/director browser (left panel) and how to put actors into workflow (right panel)

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. Drag and Drop "DDF Director" into workflow

3. Drag and Drop "String constant" into workflow

4. Change it's name to "String constant A" (Right-click -> Customize name)

5. Change it's value to "A > B" (Right-click -> Configure Actor -> value)

6. Drag and Drop "String constant" into workflow

7. Change it's name to "String constant B" (Right-click -> Customize name)

8. Change it's value to "A <= B" (Right-click -> Configure Actor -> value)

9. Drag and Drop "Constant" actor into workflow

10. Change it's name to "ConstantA" (Right-click -> Customize name)

11. Drag and Drop "Constant" actor into workflow

12. Change it's name to "ConstantB" (Right-click -> Customize name)

13. Drag and Drop "DDF Boolean Select" actor into workflow

14. Drag and Drop "Display" actor into workflow

15. Drag and Drop "Is present" actor into workflow

16. Drag and Drop "Stop" actor into workflow

17. Drag and Drop "Expression" actor into workflow

18. Add input ports to "Expression" actor (Right-click -> Configure ports) - give them names "inputa", "inputb"

19. Change the value of "Expression" actor to "inputa > inputb"

20. Add "Relation" into workflow (adding relation demo)

21. After all actors are at the workflow's area, you have to connect them

22. Connect "String constant A" actor with "true" input of "DDF Boolean select" actor

23. Connect "String constant B" actor with "false" input of "DDF Boolean select" actor

24. Connect "ConstantA" actor with "inputa" of "Expression" actor

25. Connect "ConstantB" actor with "inputb" of "Expression" actor

26. Connect "Expression" ouput port with "control" port of "DDF Boolean Select"

27. Connect output port of "DDF Boolean Select" with relation

28. Connect relation with "Display" actor

29. Connect relation with "Is present actror"

30. Connect "Is present actor" with "Stop" actor

31. press "Play" button

workflow should generate output within Display actor

If you are not sure how to proceed, you can watch demo movie at following location: movie

2.3 Loop workflow

Exercise no. 5 (approx. 5 minutes)

In this exercise you will execute simple Kepler workflow. In order to this follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. open "Loop" workflow by issuing: File -> Open and navigate to:

$KEPLER_HOME/demos/japi4hpcgrid/simple/array_loop_composite_elements.xml

3. after workflow is opened, press "Play" button

workflow should generate output within Display actor

If you are not sure how to proceed, you can watch demo movie at following location: movies

These are simple workflows presenting Kepler's main concepts. "Hello world" workflow prints simple message, "if-else" workflow shows main concepts of the loops within Kepler, while "Loop" workflow presents more sophisticated and very useful workflow for building loops - it shades some light on the concept of so called composite actors as well.

Animating workflows

In Kepler it is possible to animate workflows during execution. In order to animate workflow you have to turn on animations. You can do this by choosing: Tools -> Animate at Runtime...
Demo movie for this feature can be found at following location: animation

3. Executing GRID based workflows

In this section we will take a closer look to HPC/GRID related components within Kepler. In order to use these workflows you need VOMS proxy. This will be the first step to get HPC/GRID jobs running

3.1 Generating VOMS proxy

After installation of Java API 4 HPC/GRID you should be able to locate vomsproxy.xml workflow at following location:

$KEPLER_HOME/demos/japi4hpcgrid/common/vomsproxy.xml
VOMS proxy workflow
Exercise no. 6 (approx. 10 minutes)

In this exercise you will execute simple GRID/HPC related workflow. In order to this follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. open "vomsproxy" workflow by issuing: File -> Open and navigate to:

$KEPLER_HOME/demos/japi4hpcgrid/common/vomsproxy.xml

3. After workflow is opened you have to modify some of the parameters

3.1. Set the location of the certificate's private file (you have to modify key parameter)
3.2. Set the location of the certificate's public file (you have to modify cert parameter)
3.3. Set the output location for the voms proxy (you have to modify proxyFile parameter)

4. after setting up the parameters, press "Play" button

workflow should generate output within Display actor and should save VOMS Proxy into specified location. Since now, you can use the VOMS proxy for the latter purpose while using GRID/HPC related workflows.

If you are not sure how to proceed, you can watch demo movie at following location: [vomsproxy.ogg]

GRID certificates for tutorial session

You will be provided with GRID certificates for the itut VO as well as with password for the private key

3.1 Submitting GRID based job

VOMS proxy is required for this workflow

You have to use vomsproxy.xml workflow prior to starting GRID job submission

In order to submit GRID based job you have to open i2g_uploadsubmitdownload.xml workflow - it can be found at following location:

$KEPLER_HOME/demos/japi4hpcgrid/i2g/i2g_uploadsubmitdownload_workflow.xml
Workflow for job submission into i2g infrastructure
Exercise no. 7 (approx. 15 minutes)

In this exercise you will submit simple GRID application using Java API 4 HPC/GRID. In order to this follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. open "i2g_uploadsubmitdownload_workflow.xml" workflow by issuing: File -> Open and navigate to:

$KEPLER_HOME/demos/japi4hpcgrid/i2g/i2g_uploadsubmitdownload_workflow.xml

3. After workflow is opened you have to modify some of the parameters

3.1. Set the location of the proxy file (one that was generated using vomsproxy.xml - you have to modify proxyLocation parameter)
3.2. Set the location of the input file (this can be some sort of text file. By default there is one installed for you at: $KEPLER_HOME/demos/japi4hpcgrid/data/file.txt - you have to modify inputLocation parameter)
3.3. Set the output directory location for the output files (e.g. /tmp/output - you have to modify outputLocation parameter)

4. after setting up the parameters, press "Play" button

After started, workflow will:

  1. upload input file into GRID storage
  2. execute GRID job
  3. download job's output inot local machine

you can find your output (StdOut, StdErr) inside outputLocation directory).

If you are not sure how to proceed, you can watch demo movie at following location: []

3.2 Submitting GRID based job - slightly modified workflow

Exercise no. 8 (approx. 15 minutes)

In this exercise try to modify workflow from the previous exercise (Submitting GRID based job). Try to submit some other applications by changing Executable parameter of the workflow. Try using some other applications, e.g. '/bin/date', '/bin/echo "Test"' and so on

After setting up the parameters, press "Play" button

You can find your output (StdOut, StdErr) inside outputLocation directory).

3.3 Submitting HPC based job

VOMS proxy is required for this workflow

You have to use vomsproxy.xml workflow prior to starting GRID job submission

In order to submit HPC based job you have to open kepler_simple.xml workflow at following location:

$KEPLER_HOME/demos/japi4hpcgrid/hpc/kepler_simple.xml
Workflow for job HPC related job submission
Exercise no. 9 (approx. 10 minutes)

In this exercise you will submit simple HPC application using Java API 4 HPC/GRID. In order to this follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. open "kepler_simple.xml" workflow by issuing: File -> Open and navigate to:

$KEPLER_HOME/demos/japi4hpcgrid/hpc/kepler_simple.xml

3. After workflow is opened you have to modify some of the parameters

3.1. Set the location of the proxy file (one that was generated using vomsproxy.xml - you have to modify proxyLocation parameter)

4. after setting up the parameters, press "Play" button

After started, workflow will:

  1. submit job into HPC machine
  2. retrieve job's and show it inside Display actor
Useful Information

You can submit various job into HPC machne by changing executable parameter of the actor.

4. Concept of composite actors

Workflows presented in previous section looks as a single actor workflows. In fact, these are made of so called - composite actors. Composite actors allows you to build complex workflows and encapsulate them within an actor. This way you can easily access complex structures by simply putting single actor within workflow. Composite actors can be also exported as KAR files and put into components panel for later usage.

Workflow presenting composite actor

4.1 Building simple composite actor

Exercise no. 10 (approx. 20 minutes)

In this exercise you will create Composite actor and start the workflow. In order to achieve this goal, follow the instruction:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. in the workflow area choose: Tools -> Create composite actor

2. Add SDF director by entering: "SDF" within Search window and pressing "Search"

3. Drag and Drop SDF director into workflow area

5. Right click SDF director and choose: Configure director.

6. Set number of iterations to: 1 and Commit changes

7. Configure ports of the composite actor: Right click Composite actor -> Configure ports (add input/output ports)

8. Add String constant into workflow by typing "String constant" within Search window and pressing "Search"

9. Drag and Drop "String constant" into workflow

10. Enter value into "String constant" by double-clicking it

11. Connect "String constant" output port with "Composite actor" input port

12. Drag and Drop "Display" actor into workflow

13. Connect "Composite actor" output port with "Display" input port

14. Opent composite actor by Right click on Composite actor -> Open actor

15. Add relation into workflow by clicking "Relation button"

16. Connect input port of the actor with relation (Ctr+left click on port) - inside composite actor

17. Connect output port of the actor with relation (Ctrl+left click on port) - inside composite actor

18. Add Display actor into workflow - inside composite actor

19. Connect relation with "Display" input port (Ctrl+left click on relation)

20. Press Play button

You should see two Display windows. One was generated by the Composite actor, while the other one was generated by the main workflow.

If you are not sure how to proceed, you can watch demo movie at following location: movie

5. Executing HPC/GIRD based jobs in single workflow

Java API 4 HPC/GRID allows not only submission of the job into particular infrastructure. It also allows to mix infrastructure access, which means that you can submit both: HPC and GRID based job from within single workflow.

Exercise no. 11 (approx. 15 minutes)

In this exercise you will submit jobs into both HPC and GRID infrastructures. In order to complete this exercise, follow the instructions:

1. start Kepler application by issuing:

$KEPLER_HOME/kepler.sh

2. open "hpc_grid_preinstalled.xml" workflow by issuing: File -> Open and navigate to:

$KEPLER_HOME/demos/japi4hpcgrid/common/hpc_grid_preinstalled.xml

3. After workflow is opened you have to modify some of the parameters

3.1. Set the location of the proxy file (one that was generated using vomsproxy.xml - you have to modify proxyLocation parameter)

3.2. Set the location of the "script_date.sh" file (it should be located inside $KEPLER_HOME/demos/japi4hpcgrid/scripts/script_date.sh)

3.3. Set the location of the "gnuplot.script" file (it should be located inside $KEPLER_HOME/demos/japi4hpcgrid/scripts/gnuplot.script)

3.4. Set the location of the output directory (usually: /tmp/output)

4. after setting up the parameters, press "Play" button

After started, workflow will:

  1. submit job into HPC machine
  2. sumbit job into GRID
  3. collect output from HPC and from GRID
  4. pass the outputs as inputs into new GRID job
  5. submit job GRID that will create gnuplot's output
  6. retrieve gnuplot's output and show it within Kepler
Workflow for job HPC/GRID related job submission

6. Accessing GRID environment using Migrating Desktop

Migrating Desktop allows you to use GRID in convenient, User Friendly way while at the same times adds useful features like: file management (GridCommander), viewers for the output data, interactivity.

Migrating Desktop can be started directly from the Web browser (Java 1.6.0_16 required) - MigratingDesktop. It can be also started using following command:

$JAVA_HOME/bin/javaws http://stipa.man.poznan.pl/md/JWS/MigratingDesktop.jnlp
Exercise no. 12 (approx. 10 minutes)

In this exercise you will check job statuses of the GRIG job's using Migrating Desktop. In order to complete the task, follow instruction:

1. start Migrating Desktop using one of the methods:

1.1. use JNLP link: MigratingDesktop

1.2. Start it using javaws application from the command line

$JAVA_HOME/bin/javaws http://stipa.man.poznan.pl/md/JWS/MigratingDesktop.jnlp

2. After login window pops-up, set the certificate information (replace "user11" with you certificate information)

3. Make sure that server settings are the same as at the picture below

4. Login to the Migrating Desktop

5. After Migrating Desktop is started, run Job Monitoring Dialog, by pressing an icon at tool bar

6. You can browse the jobs that were submitted using Kepler

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.