The purpose of this tutorial is to introduce the concepts and methods for adding external software/codes to a Kepler workflow and demonstrate how to do it in practice with an example code. Whilst Kepler is designed for functionality to be implemented within Kepler using the actors available (or developing custom actors) it is often desirable to be able to call and interact with programs that are external to Kepler. For example, process and prepare data for a simulation code, run the code, then visualise the data after the code has finished. We have introduced actors that enable users to do this with HPC and Grid resources in the [Grid and HPC] tutorial but this tutorial covers the basic concepts and techniques required to interface both an external Java code and also other language codes (such as C and Fortran) from a Kepler workflow.
We will present two different methods for calling external software from Kepler, one where the user is interested in running a complete executable from Kepler and the other which allows users to execute different parts of their code in the workflow as required. They will be outlined in the following sections along with examples.
This tutorial assumes that you have basic Kepler knowledge and can use Kepler on the Gateway machine. Please see the [Basic Kepler Training] tutorial if you need more information.
By far the simplest method for running an external code or program from Kepler is to use the External Execution actor.
This actor is one of a number of Kepler actors designed to allow programs or commands to be executed from a workflow (the other main actors in this category are InteractiveShell, SSHToExecute, and UserInteractiveShell).
The ExternalExecution actor can pass values or data to the application being run and return data that can be used or displayed in the workflow. The targeted application must be available on the same computer as the workflow is being run on and may require configuration for correct operation.
As can be seen in the picture above the actor takes the command to execute as an input (or as a fixed parameter, see the image below). It can also take data input, and produce data output, error codes, and exit codes. Furthermore, the working directory (directory in which the application will run) is specified in the actor's parameters (see the image below). By default this is the current working directory that Kepler was run from. It is also possible to specify environment variables to be set in the environment the application will be run in (this are specified as name-value pairs as outlined in the parameter box).
The actor is part of the standard Kepler library and can be found under "Components/General Purpose/Unix Command" in the actor library or via a search under the Components tab. Next is a brief example of using this actor.
Whilst the above exercise has demonstrated the running of external programs using a very simple command (i.e. "ls") it the same process can be used to add any pre-compiled program which is available for running on the same machine as the Kepler workflow is being executed.
The JNI (Java Native Interface) is a programming framework that allows Java to call a native applications and libraries (and vice versa) written in languages other than Java (primarily C and C++; FORTRAN is not directly support but can be called through a C or C++ interface).
JNI was primarily designed to access native methods where functionality cannot be entirely implemented in Java or where better performance can be obtained outside Java; for instance to access platform-specific features which Java does not support or to access a library not implemented in Java. In fact many of the standard libraries used in Java make use of the JNI to provide their functionality (for instance file I/O is often implemented through such an interface). Furthermore, the JNI framework can let native code access Java objects in the same way that Java uses these objects (i.e. C can call Java as well as Java call C). A native method can create Java objects and then inspect and use these objects to perform its tasks. A native method can also inspect and use objects created by Java application code.
For our purposes we are primarily interested in using it to modify an existing application (i.e. a C code) to make it callable from a Java applications (i.e. a Kepler workflow). Whilst it is beyond the scope of this tutorial to give a full and comprehensive description of how to use and program with JNI we will need to discuss the basics of programming with JNI and Java. The following section outlines basic JNI programming and then the next section will discuss how to integrate the generated code with a Kepler workflow.
Methods or functions to be called by JNI should be implemented in separate files, and the functions to be called by Java should take the following form:
Where "Arguments" are the Java arguments declared by the Java method and the JNIEnv pointer and jobject pointer are required by the JNI framework. "env" is a structure that contains the interface to the JVM, including all the functions necessary to interact with the JVM and to work with any Java objects. Example JNI functions are converting native arrays to/from Java arrays, converting native strings to/from Java strings, instantiating objects, throwing exceptions, etc.
For example, the following code examples (in C and C++) maps to a Java class called MyString which has a method called convert and it converts a Java string to a native string (i.e. a C or C++ string):
Native data types can be mapped to/from Java data types, however for compound types such as objects, arrays and strings the native code must explicitly convert the data by calling methods in the JNIEnv.
To call JNI code from Kepler the JNI Java code needs to be re-organised from how it was described in the previous sections. We have developed an actor, called JNICaller, which wraps JNI code and allows it to be called from Kepler.
- bc_exec.xml - simple workflow that executes bc application and passes
stdin - this is exactly the same result as for jni_caller.xml in the
sense of the input and output (1+2 = 3)
- jni_caller.xml - workflow that contains actor performing JNI call
- JNIExample.tar - updated JNI Examples. I had to modify Calc example in
order to fit into JavaAPI4HPC/GRID packages structure
- pl.psnc.kepler.sandbox.jar - contains actor that is able to call Calc
native library. Fully qualified name of actor is:
One thing here. Before you start Kepler 1.0 make sure that you have
exported LD_LIBRARY_PATH that contains created JNI native code!!
Best solution for you is to install Kepler 1.0 + java api for hpc and
grid from here:
with the file I have attached.
After you open jni_caller.xml workflow you will see simple workflow that
executes Calc native code. This is exactly the same library you can use
from the CLI.