01. Introduction to Kepler - basics

Introduction

This tutorial is designed to introduce the concept of building simple workflows within Kepler-1.0/Kepler-2.0.

Kepler is a workflow engine and design platform for analyzing and modeling scientific data. Kepler provides a graphical interface and a library of pre-defined components to enable users to construct scientific workflows which can undertake a wide range of functionality. It is primarily designed to access, analyse, and visualise scientific data but can be used to construct whole programs or run pre-existing simulation codes.

Kepler builds upon the mature Ptolemy II framework, developed at the University of California, Berkeley. Kepler itself is developed and maintained by the cross-project Kepler collaboration.

The main components in a Kepler workflow are actors, which are used in a design (inherited from Ptolemy II) that separates workflow components ("actors") from workflow orchestration ("directors"), making components more easily reusable. Workflows can work at very levels of granularity, from low-level workflows (that explicitly move data around or start and monitor remote jobs, for example) to high-level workflows that interlink complex steps/actors. Actors can be reused to construct more complex actors enabling complex functionality to be encapsulated in easy to use packages. A wide range of actors are available for use and reuse.

1 Installing Kepler with Serpens add-on

Kepler-2.0 is required for this tutorial session

You can download Kepler-2.0 from the following page

https://kepler-project.org/users/downloads

In order to install Kepler and Serpens related workflows you have to do following:

cd ~
wget http://scilla.man.poznan.pl/euforia/install/serpens-demo-workflows.zip -O serpens-demo-workflows.zip
unzip serpens-demo-workflows.zip

Now you can start Kepler-2.0 application and proceed to tutorial examples.

2. Executing simple workflows

In order to execute workflow, you have to load workflow XML file into Kepler. During this tutorial session we will use following workflows:

Workflow description Location
"Hello world" $HOME/serpens/demo/workflow/basic/Hello_World.xml
"Hello world debug" $HOME/serpens/demo/workflow/basic/Hello_World_Debug.xml
"If-else-simple" $HOME/serpens/demo/workflow/basic/if_else_simple.xml
"If-else-simple-expression" $HOME/serpens/demo/workflow/basic/if_else_simple_expression.xml
"Simple-Loop" $HOME/serpens/demo/workflow/basic/simple_loop.xml
"Array-Copy-Loop" $HOME/serpens/demo/workflow/basic/array_copy_loop.xml
"Feedback-Loop" $HOME/serpens/demo/workflow/basic/array_loop_composite_repeat_feedback.xml
"Loop with relation" $HOME/serpens/demo/workflow/basic/simple_loop_relation.xml
"Loop without DDF Boolean Select" $HOME/serpens/demo/workflow/basic/loop-no-ddf-b-s.xml
"Loop based on variables" $HOME/serpens/demo/workflow/basic/loop-variables.xml

2.1 Hello world workflow

2.1.1 Using existing "Hello world" workflow
After this exercise you will:
  • know how to start Kepler
  • know how to load simple workflow
  • know how to execute workflow
  • know how to animate workflow
Exercise no. 1 (approx. 10 min)

Film available: http://www.youtube.com/watch?v=1xsPH6Mnzx0
In this exercise you will execute simple Kepler workflow. In order to this follow the instructions:

1. Start Kepler application by issuing:

cd ~/Kepler-2.0
./kepler.sh

2. Open "Hello world" workflow by issuing: File -> Open and navigate to:

$HOME/serpens/demo/workflow/basic/Hello_World.xml

3. After workflow is opened, press "Play" button

workflow should generate output within Display actor

Animating workflows
In Kepler it is possible to animate workflows during execution. In order to animate workflow you have to turn on animations. You can do this by choosing: Tools -> Animate at Runtime...
Demo movie for this feature can be found at following location: animation
2.1.2 Using existing "Hello world - with debug" workflow
After this exercise you will:
  • know how to start Kepler
  • know how to load simple workflow
  • know how to execute workflow
  • know how to listen to the actor
Exercise no. 2 (approx. 10 min)

Film available: http://www.youtube.com/watch?v=EVGSXC4kcks
In this exercise you will execute simple Kepler workflow with Debug information. In order to this follow the instructions:

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2. Open "Hello world debug" workflow by issuing: File -> Open and navigate to:

$HOME/serpens/demo/workflow/basic/Hello_World_Debug.xml

3. After workflow is opened, "Right-click" Expression actor and choose "Listen to actor"

4. Press "Play" button

Workflow should generate output within Display actor and should print debug information generated by Expression actor

2.1.3 Building "Hello world" from the scratch
After this exercise you will:
  • know how to start Kepler
  • know how to build simple workflow
  • know how to connect elements
  • know how to add elements to the workflow
  • know how to search for the actors within Kepler's library
Exercise no. 3 (approx. 15)

Film available: http://www.youtube.com/watch?v=DXXYnuDjnWw

In this exercise you will build simple "Hello World" workflow and execute it. In order to get this task done, follow the instructions:

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2. Type in "SDF" into "Search" field and press "Search" button

3. Drag and Drop SDF director into workflow

4. Right-click on SDF director and choose "Configure Director"

5. Set number of iterations to "1"

6. Type in "String" into "Search" field and press "Search" button

7. Add "String Constant" actor to the workflow

8. Right-click "String Constant" actor and choose "Configure Actor"

9. Type "Hello world!" into "value" field

10. Commit changes

11. Type in "Display" into "Search" field and press "Search" button

12. Add "Display" actor into workflow

13. Connect "String Constant" actor with "Display" actor

Intermediate results

14. after workflow is ready, press "Play" button

workflow should generate output within Display actor

2.2 Relations, Paths and Synchronization

After this exercise you will:
  • know how to add elements into workflow
  • know hot to use expressions
  • know how to synchronize workflow's execution
  • know how to use parametrs
  • know how to use relations
Exercise no. 4 (approx. 20 minutes)

Film available: http://www.youtube.com/watch?v=OCO9L5MzUrM

A. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

1. Add DDF director into workflow
2. Add Constant into workflow and set it's value to "true" (double click Constant and enter "true")
3. Set Constant "firingCountLimit" to "1" (Right Click -> Configure Actor -> firingCountLimit Text Field)
4. Add Relation next to the Constant
5. Connect Relation and Constant::output
6. Add Parameter and rename it to "a" (Right clicke -> Customize name)
7. Set value of a to "1" (double click a)
8. Add Parameter and rename it to "b" (Right clicke -> Customize name)
9. Set value of b to "2" (double click b)
10. Add Constant into workflow and rename it to "Send a to output"
11. Set Send a to output value to "a"
12. Add Constant into workflow and rename it "Send b to output"
13. Set *Send b to output" value to "b"
14. Connect Send a to output::trigger" with *Relation
15. Connect Send b to output::trigger" with *Relation
16. Add Relation to workflow and connect it with Send a to output
17. Add Relation to workflow and connect it with Send b to output
18. Add Display to workflow and connect relation connected to Send a to output
19. Set Display "Display name" (Right click -> Customize Name) to "Value a"
20. Add Display to workflow and connect it with other relation
21. Set Display "Display name" (Right click -> Customize Name) to "Value b"
22. Add Expression to the workflow
23. Add input port input_a to the Expression (Right click -> Configure Ports -> Add, select checkbox "in")
24. Add input port input_b to the Expression (Right click -> Configure Ports -> Add, select checkbox "in")
25. Connect Expression::input_a with relation bound to Send a to output
26. Connect Expression::input_b with relation bound to Send b to output
27. Add Display to the workflow and set it's "Display name" to "Result"
28. Connect Result::input with *Expression::output"
29. Set Expression value to "a+b" (Double click Expression)

At this point your workflow should be similar to the one below

Intermediate results

30. Execute workflow

Simple modification in order to make Kepler workflow fail

31. Set Expression value to "a/b"
32. Set b value to "0"

2.3 If-else workflow

2.2.1 Using existing "if-else" workflow
After this exercise you will:
  • know how to use different paths for data flow
  • know how to split workflow execution path
  • know how to use Boolean Switch actor
Exercise no. 5 (approx. 10 minutes)

Film available: http://www.youtube.com/watch?v=rr03bekyiDU

In this exercise you will execute simple Kepler workflow. In order to this follow the instructions:

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2. Open "If-else" workflow by issuing: File -> Open and navigate to:

$HOME/serpens/demo/workflow/basic/if_else_simple.xml

3. After workflow is opened, press "Play" button

workflow should generate output within Display actor

2.2.2 Building "if-else" from the scratch
After this exercise you will:
  • know how to use different paths for data flow
  • know how to split workflow execution path
  • know how to use Boolean Switch actor
Exercise no. 6 (approx. 20 minutes)

Film available: http://www.youtube.com/watch?v=3M7IFyzSTAY

In this exercise you will build "if-else" workflow.

You should complete previous examples before starting this one
In this example it is assumed that you already know how to use actor/director browser (left panel) and how to put actors into workflow (right panel)

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2. Drag and Drop "DDF Director" into workflow
3. Drag and Drop "String Constant" actor into workflow
4. Change it's name to "String Hello" (Right-click -> Custimize name)
5. Change it's value to "Hello world will test if-else" (Right-click -> Configure Actor -> value)
6. Change it's firingCountLimit to "1" (Right-click -> Configure Actor -> firingCountLimit)
7. Drag and Drop "Parameter" actor into workflow
8. Change it's name to "a" (Right-click -> Customize name)
9. Change it's value to "1" (Double click -> value)
10. Drag and Drop "Parameter" actor into workflow
11. Change it's name to "b" (Right-click -> Customize name)
12. Change it's value to "2" (Double click -> value)
13. Drag and Drop "Boolean Switch" actor into workflow
14. Drag and Drop "Display" actor into workflow next to "Boolean Switch" actor
15. Change it's name to "Display if"
16. Drag and Drop "Display" actor into workflow below "Display if" actor
17. Change it's name to "Display else"
18. Drag and Drop "Constant" actor into workflow below "Boolean Switch" actor
19. Change it's value to "a < b" (Right-click -> Configure Actor -> value)
20. Change it's firingCountLimit to "1" (Right-click -> Configure Actor -> firingCountLimit)
21. After all actors are at the workflow's area, you have to connect them
22. Connect Boolean Switch::trueOutput with Display if::input
23. Connect Boolean Switch::falseOutput with Display else::input
24. Connect Boolean Switch::input with String Hello::output
25. Connect Boolean Switch::control with Constant::output

At this point your workflow should be similar to the one below

Intermediate results

26. press "Play" button

workflow should generate output within "Display else" actor

27. Change value of Constant to "a > b" and execute workflow once again
28. Save the workflow (e.g. as ~/my_workflow.xml) - we will need it in next excersise

2.2.2 Building "if-else-expression" from the scratch
After this exercise you will:
  • know how to use different paths for data flow
  • know how to split workflow execution path
  • know how to use Boolean Switch actor
  • know how to use Expression
  • know how to use data flowing within workflow
Exercise no. 7 (approx. 20 minutes)

Film available: http://www.youtube.com/watch?v=qC6eVPXW4Fs

In this exercise you will build "if-else-expression" workflow.

You should complete previous examples before starting this one
In this example it is assumed that you already know how to use actor/director browser (left panel) and how to put actors into workflow (right panel)

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2. Load workflow that you have previously saved (~/my_workflow.xml) or open workflow at following location:

$HOME/serpens/demo/workflow/basic/if_else_simple.xml

3. Remove link between Display if and Boolean Switch (select link and press "Delete" or choose Edit -> Delete)

4. Remove link between Display else and Boolean Switch

5. Add Expression between Display if and Boolean Switch

6. Set Expression Display name to "Expression if"

7. Add input port in into Expression if

8. Set Expression if value to

in + " - this was added by Expression if"

remember to copy " as well!

9. Connect Expression if::in with Boolean Switch::trueOutput

10. Connect Expression if::output with Display if::input

11. Add Expression between Display else and Boolean Switch

12. Set Expression Display name to "Expression else"

13. Add input port in into Expression else

14. Set Expression else value to

in + " - this was added by Expression else"

remember to copy " as well!

15. Connect Expression else::in with Boolean Switch::falseOutput

16. Connect Expression else::output with Display else::input

At this point your workflow should be similar to the one below

Intermediate results

17. press "Play" button

workflow should generate output within "Display if" actor

18. Change value of Constant to "a < b" and execute workflow once again

3 Loops within Kepler

In this section of tutorial we will go through basic concepts of looping within Kepler. We will execute simple loop, build it from the scratch and, at the end, we will go through more complex examples of loops.

3.1 Executing simple loop example (classic)

After this exercise you will:
  • know how to build simple loops
  • know how to use SampleDelay actor
  • know how to create loop condition checks
  • know difference between SDF and DDF Directors
Exercise no. 8 (approx. 15 minutes)

Film available: http://www.youtube.com/watch?v=fJlV7Jd30cQ

In this exercise you will execute simple loop example. In order to this follow the instructions:

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2 Open workflow

$HOME/serpens/demo/workflow/basic/simple_loop.xml

You should see following workflow loaded into Kepler

3. After workflow is loaded, execute it

workflow should generate output within Display actor

4. You can change Constant values and see what happens after you start workflow again

3.2 Executing simple loop example (using relation instead of SampleDelay)

After this exercise you will:
  • know how to build simple loops
  • know how to create loop condition checks
Exercise no. 9 (approx. 15 minutes)

In this exercise you will execute simple loop example. In order to this follow the instructions:

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2 Open workflow

$HOME/serpens/demo/workflow/basic/simple_loop_relation.xml

You should see following workflow loaded into Kepler

3. After workflow is loaded, execute it

workflow should generate output within Display actor

4. You can change Constant values and see what happens after you start workflow again

3.3 Executing simple loop example (using parameters and Variable Setter)

After this exercise you will:
  • know how to build simple loops
  • know how to use Variable Setter actor
  • know how to create loop condition checks
Exercise no. 10 (approx. 15 minutes)

In this exercise you will execute simple loop example. In order to this follow the instructions:

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2 Open workflow

$HOME/serpens/demo/workflow/basic/loop-variables.xml

You should see following workflow loaded into Kepler

3. After workflow is loaded, execute it

workflow should generate output within Display actor

4. You can change Constant values and see what happens after you start workflow again

3.4 Executing simple loop example (without DDF Boolean Select actor)

After this exercise you will:
  • know how to build simple loops
  • know how to create loop condition checks
Exercise no. 11 (approx. 15 minutes)

In this exercise you will execute simple loop example. In order to this follow the instructions:

1. If Kepler is not already running start it by issuing:

cd ~/Kepler-2.0
./kepler.sh

2 Open workflow

$HOME/serpens/demo/workflow/basic/loop-no-ddf-b-s.xml

You should see following workflow loaded into Kepler

3. After workflow is loaded, execute it

workflow should generate output within Display actor

4. You can change Constant values and see what happens after you start workflow again

3.5 Building simple loop from the scratch

After this exercise you will:
  • know how to build simple loops
  • know how to use SampleDelay actor
  • know how to create loop condition checks
  • know difference between DDF and SDF directors
Exercise no. 12 (approx. 30 minutes)

Film available: http://www.youtube.com/watch?v=oYdOYnK7WI4

In this exercise you will build simple loop. In order to this follow the instructions:

1. Start Kepler application by issuing:

cd ~/Kepler-1.0.0
./kepler.sh

2. Add DDF director into workflow

3. Add Constant into workflow, set it's value to 5

4. Add DDF Boolean Select actor to the workflow

5. Connect Constant::output with DDF Boolean Select::falseInput

6. Add Relation next to DDF Boolean Select

7. Connect Relation (we will call it Relation A) with DDF Boolean Select::output

8. Add Expression actor to the workflow (next to Relation A)

9. Add input port in into Expression

10. Connect Relation A with Expression::in

11. Set Expression value to

 in > 0 

12. Add Relation next to Expression (we will call it Relation B)

13. Connect Expression::output with Relation B

Intermediate result

14. Add SampleDelay actor to workflow

15. Change SampleDelay::input port direction to EAST (Right click -> Configure Ports -> Direction)

16. Change SampleDelay::output port direction to WEST (Right click -> Configure Ports -> Direction)

17. Connect SampleDelay::output with DDF Boolean Select::control

18. Connect SampleDelay::input with Relation B

19. Set SampleDelay value to

 {false} 

20. Add Boolean Switch next to Relation B

21. Connect Boolean Switch::control with Relation B

22. Connect Boolean Switch::input with Relation A

Intermediate result

23. Add Expression into workflow and set it's name to Decrease counter

24. Add input port in into Decrease counter and set it's Direction to EAST

25. Set output port Direction to WEST

26. Set Expression value to

 in - 1 

27. Connect Decrease counter::output with DDF Boolean Select::trueInput

28. Connect Decrease counter::input with Boolean Select::trueOutput

29. Add Is Present next to Boolean Switch

30. Connect Is Present::input with Boolean Switch::falseOutput

31. Add Stop next to Is Present

32. Connect Stop::input with Is Present::output

33. Add Display next to Relation A

34. Connect Display::input with Relation A

Intermediate result

35. After workflow is opened, press "Play" button

36. You can change Constant values and see what happens next time you start workflow

3.6 Executing advanced loop workflow (Composite loop + Repeat)

After this exercise you will:
  • know how to utilize loop concept
  • know how to build advanced loop workflows
  • know how to use Repeat actor
  • know how to set value of the parameter
Exercise no. 13 (approx. 15 minutes)

Film available: http://www.youtube.com/watch?v=muhBH7jM5dU

In this exercise you will execute advanced loop workflow. In order to this follow the instructions:

1. Start Kepler application by issuing:

cd ~/Kepler-1.0.0
./kepler.sh

2 Open workflow

$HOME/serpens/demo/workflow/basic/array_loop_copy_files.xml

3. After workflow is loaded

You can open composite actor by right clicking it and choosing "Open"

4. You can start workflow by pressing Play button

workflow should copy input files from/to

$HOME/serpens/demo/data -> $HOME/serpens/demo/output

You can open terminal and verify it's execution results

ls -la ~/serpens/demo/data
ls -la ~/serpens/demo/output

3.7 Executing advanced loop workflow (Composite loop + feedback)

After this exercise you will:
  • know how to utilize loop concept
  • know how to build advanced loop workflows
  • know how to use Repeat actor
  • know how to set value of the parameter
Exercise no. 14 (approx. 15 minutes)

Film available: http://www.youtube.com/watch?v=4xjLcI776vg

In this exercise you will execute advanced loop workflow with feedback. In order to this follow the instructions:

1. Start Kepler application by issuing:

cd ~/Kepler-1.0.0
./kepler.sh

2 Open workflow

$HOME/serpens/demo/workflow/basic/array_loop_composite_repeat_feedback.xml

3. After workflow is loaded

open composite actor by right clicking it and choosing "Open"

4. You can execute it by pressing Play button

workflow should generate output within Display actor

Animating workflows
In Kepler it is possible to animate workflows during execution. In order to animate workflow you have to turn on animations. You can do this by choosing: Tools -> Animate at Runtime...
Demo movie for this feature can be found at following location: animation

3.7 Creating a loop using PythonScript actor

After this exercise you will:
  • know how to use Python script inside a Kepler actor
  • know how to create a generic loop using PythonScript actor
Exercise no. 15 (approx. 15 minutes)

In this exercise you will create a simple loop in Python and put it inside a special Kepler actor. In order to this follow the instructions:

  1. Start Kepler application by issuing:
    cd ~/Kepler-1.0.0
    ./kepler.sh
    
  2. Instantiate a PythonScript actor by choosing menu Tools -> Instantiate Component and setting as Class name a value ptolemy.actor.lib.python.PythonScript.
  3. This actor starts with zero ports. They need to be added manually. Please right-click on PythonScript and choose Configure Ports.
  4. Add an input port named in and output port named out.
    Kepler's automatic type resolver may not correctly infer types of PythonScript ports due to dynamic features of Python programming language. This may lead to errors and unexpected behaviour. Thus you need to specify these types explicitly. For this tutorial, please set type of in to int and type of out to arrayType(int).
  5. By default Kepler initialises the script parameter of this actor to be of type Line. To develop a script in Python, it needs to be changed. Please right-click on PythonScript and choose Configure Actor. Go to Preferences and select expert mode.
  6. Close the window with actor's preferences and once again start with right-clicking and choosing Configure Actor. Again choose Preferences and change type of script parameter to Text.
  7. Now you can see a Python code displayed in several lines. Some remarks here:
    1. Python is a dynamic language, so no typecasting takes place,
    2. Do not declare any constructor.
    3. You only need to fill the fire() method.
    4. You can assume that the configured ports are already instantiated (ie. you can use names in and out to work with actor's ports)
  8. Let's assume a following problem to solve. The actor receives a number which will be interpreted as iteration count. In each i-th iteration, the actor will output i2. Example: in = 4, out = {0, 1, 4, 9}.
  9. In Python, the following script will do this:
    import ptolemy.data
    
    class Main:
        def fire(self):
            # read value of input token
            val = self.in.get(0).intValue()
            arr = []
            for i in range(val):
                # create a new IntToken with each value
                arr.append(ptolemy.data.IntToken(i**2))
            # send an ArrayToken with array of values
            self.out.send(0, ptolemy.data.ArrayToken(arr))
            return
    
  10. You can now instantiate Constant actor. Set its firingCountLimit to 1 and value to 4. Connect it with in port of PythonScript.
  11. Instantiate also a Display actor and connect PythonScript's out with it.
  12. Finally add an SDF actor and execute the workflow.
  13. You will see {0, 1, 4, 9}. Now you can change the input value 4 to some other one. Or you can change the actor source code to execute a different task.

3.8 Creating a time-loop with series plotting

After this exercise you will:
  • know how to provide data from the loop to the plotting actor
Exercise no. 16 (approx. 15 minutes)

In this exercise you will create a simple loop containing some potentially time-consuming operations which will be plotted live. In order to do this follow the instructions:

  1. Start Kepler application by issuing:
    cd ~/Kepler-1.0.0
    ./kepler.sh
    
  2. Put DDF Director.
  3. Put Constant actor and set its firingCountLimit to 1. This will be the starting source for the example workflow.
  4. Put a relation symbol (Ctrl+click) next to the Constant and link them.
  5. Put an Expression actor and add an input port named in. Let's set the expression to in * in which indicates that we will plot the square function. Then connect the relation with in port.
  6. Put an XYPlotter actor and connect its inputX with the relation symbol and inputY with Expression actor's output port.
  7. Your workflow should look like the one presented below. It applies some function to input data and plots it. However, it works for a single value now. We need to create a loop.
  8. Put another Expression actor. Add an input port named in, set expression to in + 1 and connect the input port with relation. This is responsible for the step of loop.
  9. Create another relation symbol next to this Expression actor and connect them.
  10. Put next Expression actor. Again add an input port named in, set expression to in < 10 and connect the input port with the just created relation. This is responsible for loop termination when it reaches specific point.
  11. Put Boolean Switch actor, connect its input port to the in + 1 expression and its control port to the in < 10 expression.
  12. Now we want to simulate the time-consuming behaviour, so we are going to add an artificial sleep time. To do this, you need to choose from menu Tools -> Instantiate Component and set as Class name a value ptolemy.actor.lib.Sleep. A new Sleep actor will appear. It's purpose is to grab some input, wait for the specified amount of time and then send the data. For this workflow, please set its sleepTime to 1000 (the unit here is milliseconds, so we will simulate one second of time-consuming operations).
  13. Connect Boolean Switch trueOutput port with Sleep's input, and Sleep's output to the relation symbol at the beginning of the loop. Your workflow should look like the one below. You can run it and you will see that the output is plotted live and updated every second (ie. every time it receives new data).
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.