05. Adding Existing Codes to a Workflow

Adding simulation codes to a workflow in Kepler

Introduction

The purpose of this tutorial is to introduce the concepts and methods for adding external software/codes to a Kepler workflow and demonstrate how to do it in practice with an example code. Whilst Kepler is designed for functionality to be implemented within Kepler using the actors available (or developing custom actors) it is often desirable to be able to call and interact with programs that are external to Kepler. For example, process and prepare data for a simulation code, run the code, then visualise the data after the code has finished. We have introduced actors that enable users to do this with HPC and Grid resources in the [Grid and HPC] tutorial but this tutorial covers the basic concepts and techniques required to interface both an external Java code and also other language codes (such as C and Fortran) from a Kepler workflow.

We will present two different methods for calling external software from Kepler, one where the user is interested in running a complete executable from Kepler and the other which allows users to execute different parts of their code in the workflow as required. They will be outlined in the following sections along with examples.

Requirements
This tutorial assumes that you have basic Kepler knowledge and can use Kepler on the Gateway machine. Please see the [Basic Kepler Training] tutorial if you need more information.

External Execution

By far the simplest method for running an external code or program from Kepler is to use the External Execution actor.
This actor is one of a number of Kepler actors designed to allow programs or commands to be executed from a workflow (the other main actors in this category are InteractiveShell, SSHToExecute, and UserInteractiveShell).

The ExternalExecution actor can pass values or data to the application being run and return data that can be used or displayed in the workflow. The targeted application must be available on the same computer as the workflow is being run on and may require configuration for correct operation.

ExternalExecution Actor

As can be seen in the picture above the actor takes the command to execute as an input (or as a fixed parameter, see the image below). It can also take data input, and produce data output, error codes, and exit codes. Furthermore, the working directory (directory in which the application will run) is specified in the actor's parameters (see the image below). By default this is the current working directory that Kepler was run from. It is also possible to specify environment variables to be set in the environment the application will be run in (this are specified as name-value pairs as outlined in the parameter box).

ExternalExecution Actor Parameters

The actor is part of the standard Kepler library and can be found under "Components/General Purpose/Unix Command" in the actor library or via a search under the Components tab. Next is a brief example of using this actor.

External Execution Example

This exercise will build a workflow that calls an external command. For this example were are using the shell command "ls" which lists files in a directory. Following these steps to create and use the workflow:

1. Start Kepler and locate the External Execution actor by typing "External Execution" into the "Search" field and press "Search" button

2. Drag and Drop the External Execution actor on to the workflow

3. Add a "Display" actor to the workflow and content all the output ports of the External Execution actor to the input port of the "Display" actor.

4. Search for a "String Constant" actor and add it to the workflow.

5. Right-click the "String Constant" actor and choose "Configure Actor"

6. Type "ls -l" into the "value" field

7. Connect the "String Constant" actor output to the "Command" input port (the middle port) of the External Execution actor

8. Add a "StringParameter" actor to the workflow.

9. Right-click the "StringParameter" actor and select "Customize Name". This allows you to change the name of the actor. Change the text in both the "New name" and "Display name" fields to read "WorkingDirectory".

10. Double click with the left mouse button the "StringParameter" actor and select enter the text "$CWD" into the text box.

11. Now we need to configure the External Execution actor so it takes the data from the "WorkingDirectory" actor and runs properly in the workflow. Right-click the External Execution actor and choose "Configure Actor". Type "$WorkingDirectory" into the "Directory" field and "1" into the "firingCountLimit" field (to only run the command once).

12. Finally add an "SDF Director" to the workflow and it should be ready to run.

The final workflow should look something like the one pictured below.

Example External Execution Workflow

13. Run the workflow and observe the output produced. Try changing the value of the "WorkingDirectory" parameter and re-running the workflow to see what effect that has on the results.

Whilst the above exercise has demonstrated the running of external programs using a very simple command (i.e. "ls") it the same process can be used to add any pre-compiled program which is available for running on the same machine as the Kepler workflow is being executed.

JNI

The JNI (Java Native Interface) is a programming framework that allows Java to call a native applications and libraries (and vice versa) written in languages other than Java (primarily C and C++; FORTRAN is not directly support but can be called through a C or C++ interface).

JNI was primarily designed to access native methods where functionality cannot be entirely implemented in Java or where better performance can be obtained outside Java; for instance to access platform-specific features which Java does not support or to access a library not implemented in Java. In fact many of the standard libraries used in Java make use of the JNI to provide their functionality (for instance file I/O is often implemented through such an interface). Furthermore, the JNI framework can let native code access Java objects in the same way that Java uses these objects (i.e. C can call Java as well as Java call C). A native method can create Java objects and then inspect and use these objects to perform its tasks. A native method can also inspect and use objects created by Java application code.

For our purposes we are primarily interested in using it to modify an existing application (i.e. a C code) to make it callable from a Java applications (i.e. a Kepler workflow). Whilst it is beyond the scope of this tutorial to give a full and comprehensive description of how to use and program with JNI we will need to discuss the basics of programming with JNI and Java. The following section outlines basic JNI programming and then the next section will discuss how to integrate the generated code with a Kepler workflow.

JNI Interface

Methods or functions to be called by JNI should be implemented in separate files, and the functions to be called by Java should take the following form:

 JNIEXPORT void JNICALL Java_ClassName_MethodName(JNIEnv *env, jobject obj, Arguments)
 {
     Function Code
 }

Where "Arguments" are the Java arguments declared by the Java method and the JNIEnv pointer and jobject pointer are required by the JNI framework. "env" is a structure that contains the interface to the JVM, including all the functions necessary to interact with the JVM and to work with any Java objects. Example JNI functions are converting native arrays to/from Java arrays, converting native strings to/from Java strings, instantiating objects, throwing exceptions, etc.

For example, the following code examples (in C and C++) maps to a Java class called MyString which has a method called convert and it converts a Java string to a native string (i.e. a C or C++ string):

 //C++ code
 JNIEXPORT void JNICALL Java_MyString_convert(JNIEnv *env, jobject obj, jstring javaString)
 {
     //Get the native string from javaString
     const char *nativeString = env->GetStringUTFChars(javaString, 0);
 
     //Do something with the nativeString
 
     env->ReleaseStringUTFChars(javaString, nativeString);
 }

 /*C code*/
 JNIEXPORT void JNICALL Java_MyString_convert(JNIEnv *env, jobject obj, jstring javaString)
 {
     /*Get the native string from javaString*/
     const char *nativeString = (*env)->GetStringUTFChars(env, javaString, 0);
 
     /*Do something with the nativeString*/
 
     (*env)->ReleaseStringUTFChars(env, javaString, nativeString);
 }

Native data types can be mapped to/from Java data types, however for compound types such as objects, arrays and strings the native code must explicitly convert the data by calling methods in the JNIEnv.

Simple JNI Example

The code below implements a function which adds two numbers together. The function is implemented in C and called from Java. It requires three separate files, a C file which contains the C program and two Java files which contain a class that interfaces with the C code and a Main class. Here we are dealing with a simple C to Java example with JNI, not calling C from Kepler (which will be dealt with in the next example).

#include <stdio.h>

JNIEXPORT jint JNICALL Java_Calc_sum
  (JNIEnv *, jobject, jint, jint);

JNIEXPORT jint JNICALL Java_Calc_sum(JNIEnv *env, jobject obj, jint a, jint b) {
        return a+b;
}
Calc.c
class Calc {

  public native int sum(int a, int b);
  
  static {
    System.loadLibrary("Calc");
  }
}
Calc.java
public class Main {
  public static void main(String[] args) {
    System.out.println("library: " + System.getProperty("java.library.path"));
    Calc calc = new Calc();
    int sum = calc.sum(1,2);
    System.out.println("Sum from calc: " + sum);
  }
}
Main.java

1. To undertake this exercise download this [code]. Extract the source code by running the following command:

tar xzvf JNIExamples.tar.gz

This creates a directory which contains two examples, the one outlined in the code above (in the directory JNIExamples/Calc) and another which prints out text to the screen (in the directory JNIExamples/Simple).

2. For each example follow these steps in the directory of the example:

javac *.java (this builds the java code)
make clean
make (this builds the C code into a library which can be called from Java)
setenv LD_LIBRARY_PATH . (this tells the Java where to find the C library you've just created).
java Main (this executes the code)

3. Compile and run both the examples. Examine the source code and investigate the matching between the C and Java.

JNI and Kepler

To call JNI code from Kepler the JNI Java code needs to be re-organised from how it was described in the previous sections. We have developed an actor, called JNICaller, which wraps JNI code and allows it to be called from Kepler.

  • bc_exec.xml - simple workflow that executes bc application and passes
    stdin - this is exactly the same result as for jni_caller.xml in the
    sense of the input and output (1+2 = 3)
  • jni_caller.xml - workflow that contains actor performing JNI call
  • JNIExample.tar - updated JNI Examples. I had to modify Calc example in
    order to fit into JavaAPI4HPC/GRID packages structure
insideCalc you have to compile classes slightly different when using
CLI. And make sure to compile them before you start Kepler 1.0 with
jni_caller.xml workflow.

cd ~/JNIExample/Calc
export JAVA_HOME=/path_to_java
export PATH=$PATH:$JAVA_HOME/bin
javac pl/psnc/kepler/sandbox/actor/*.java
make
export LD_LIBRARY_PATH=`pwd`
java pl/psnc/kepler/sandbox/actor/Main


Open the jni_caller.xml workflow in Kepler
  • pl.psnc.kepler.sandbox.jar - contains actor that is able to call Calc
    native library. Fully qualified name of actor is:
    pl.psnc.kepler.sandbox.actor.JNICaller

One thing here. Before you start Kepler 1.0 make sure that you have
exported LD_LIBRARY_PATH that contains created JNI native code!!
(libCalc.so)

Best solution for you is to install Kepler 1.0 + java api for hpc and
grid from here:

~/Kepler-1.0.0/lib/jar/euforia/pl.psnc.kepler.sandbox.jar

with the file I have attached.

After you open jni_caller.xml workflow you will see simple workflow that
executes Calc native code. This is exactly the same library you can use
from the CLI.

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.