Scripts

Scripts

JEM has several scripts, used to install and/or to manage JEM jobs.

Node Scripts

Here's a list of the various scripts, with a short description. All scripts are available for Windows (.cmd) and Linux/*nix (.sh) Operating Systems.

1. jem_createNode: node creation script
2. jem: it can do many things, depending on argument:

• console: Node startup script
• install: Installs this specific node as a service
• start: Starts the service of this specific node
• stop: Stops the service of this specific node
• remove: Uninstalls the service of this specific node

First of all, you have to create a node, running the jem_createNode script. It accepts only a parameter, which is the properties file for the installation.

org.pepstock.jem.commands.CreateNode passing the following arguments:

jem_createNode -properties propertiesUrl

where propertiesUrl is the URL of the properties file for the new node definition. It can reside on a shared NAS or on a HTTP server, thus making possible to install new nodes in unattended mode.

Here is the list of the properties used to create a new node:

Property name	Description	Mandatory/Optional	Default
jem.environment.name	Name of the Environment, must be the equal to Hazelcast Group name. Used to create the Environment folder (if it doesn't exist) and in JEM node configuration, of course.	mandatory
jem.environment.password	The password used to connect to the cluster	mandatory
jem.node.name	Name of the JEM node. Used to create the node folder, it's automatically calculated incrementing by 1 the already defined nodes (nodes' names always ends with a number, e.g. node000)	optional	node000
jem.domain	Node Domain name	optional
jem.affinity	Node Affinity name. In case of multiple Affinities, it's a comma-separated list of values (e.g. aff1,aff2)	optional

Global file system properties

Property name	Description	Mandatory/Optional	Default
jem.ouput	Path where all job's output files are stored. NOTE: it must exist in the file system and be mounted in a shared file system accessible from all JEM nodes. In Ant JCL can be referenced as ${jem.output}	mandatory
jem.data	Path where the datasets and files are stored. NOTE: it must exist in the file system and be mounted in a shared file system accessible from all JEM nodes. In Ant JCL it can be referenced as ${jem.data}	mandatory
jem.source	Path where all the files that can be considered sources and that can be called at runtime are stored (typically, JCL sources and INCLUDE files). NOTE: it must exist in the file system and be mounted in a shared file system accessible from all JEM nodes. In Ant JCL it can be referenced as ${jem.source}	mandatory
jem.binary	Path where all the executable files (e.g. executable programs or scripts) called at runtime by job execution are stored. NOTE: it must exist in the file system and be mounted in a shared file system accessible from all JEM nodes. In Ant JCL it can be referenced as ${jem.binary}	mandatory
jem.classpath	Path where all the Java libraries (e.g. JAR files, zip, and so on) needed for the jobs' execution are stored. NOTE: it must exist in the file system and be mounted in a shared file system accessible from all JEM nodes. In Ant JCL it can be referenced as ${jem.classpath}	mandatory
jem.library	Path where all the native system libraries (e.g. .dll, .so) needed by the executable files in the jem.binary folder are stored. NOTE: it must exist in the file system and be mounted in a shared file system accessible from all JEM nodes. In Ant JCL it can be referenced as ${jem.library}	mandatory
jem.persistence	Path of the JEM database files for the persistence, where JEM clustered maps are stored in. NOTE: it must exist in the file system and be mounted in a shared file system accessible from all JEM nodes.	mandatory

Node login security properties

Property name	Description	Mandatory/Optional	Default
jem.login.protocol.enable	Indicate if jem login protocol will be enabled. The jem login protocol permit to accept only nodes that share a common symmetric key and clients launched by user with a known X509 certificate	optional	false
jem.keystore.name	Is the prefix of the name of the keystore file containing the symmetric key for login phase purpose. The complete name will be [jem.keystore.name][jem.environemnt.name].keystore. The file will be placed in the persistence path	optional
jem.keystore.pwd	Is the password for the keystores	optional
jem.crypt.key.pwd	Is the password for the symmetric key used for encryption and decryption	optional

Persistence properties

Property name	Description	Mandatory/Optional	Default
jem.db.driver	JDBC class driver to use to connect to database to store (and then persist) all JEM entities	mandatory
jem.db.url	JDBC url to use to connect to database to store (and then persist) all JEM entities	mandatory
jem.db.user	User to use to connect to database to store (and then persist) all JEM entities	mandatory
jem.db.password	Password of user to use to connect to database to store (and then persist) all JEM entities	mandatory

Hazelcast optional properties

Property name	Description	Mandatory/Optional	Default
jem.multicast.group	Multicast-group address for Hazelcast	optional	233.0.0.1
jem.multicast.port	Multicast-port for Hazelcast cluster	optional	54327
jem.port.autoincrement	Autoincrement-port for Hazelcast member	optional	5710
jem.interface	Network interface that Hazelcast member must use for its network actions	optional	null

To start a JEM node you must run jem script, which calls org.pepstock.jem.node.Main class with the following system properties:

• -Djem.config=[JEM node configuration file] (mandatory) - JEM node configuration file
• -Dhazelcast.config=[Hazelcast configuration file] (mandatory) - Hazelcast configuration file
• -Dlog4j.config=[Log4j configuration file] (optional) - Apache Log4j configuration file

Submitting jobs

The jem_submit script is used to submit a job (defined by its JCL). It takes the following input parameters:

jem_submit -Djem.custom.property=value -password clusterPassword -jcl jclFile [-type jclType] -host nodeURL [-wait true/false] [-printOutput true/false] [-privateKey pathToPrivateKey] [-privateKeyPwd privateKeyPassword]

where:

• jem.custom.property=value could be a set of properties which will be included inside of Job entity and you can use in job listeners or inside of JCL as properties. ONLY the property with name starting with jem.custom. prefix will be managed,otherwise they will be ignored.
• jclFile is the JCL which describes the job (mandatory). Could be a URL or a File path. See here how to use the special JEM URL
• clusterPassword is the Hazelcast password (jem.environment.password) needed to connect to the cluster (optional, if not specified, will be asked via console)
• jclType is the type of JCL (optional)
• nodeURL is the URL of the web application), used to receive the list of cluster's members (mandatory)
• wait is optional argument and if false the submit process ends immediately, without waiting for the end of the job. Default is true.
• printOutput is optional argument and if true the submit process catches all standard output and error of job execution and displays them. Default is false.
• pathToPrivateKey if login security is anabled is the private user pem key used to connect to the cluster (see Cluster_Security).
• privateKeyPassword if the private user pem key is protected by password

To submit a job from a server running a JEM node, the jem_localhost_submit script is used. It takes the following input parameters:

jem_localhost_submit -Djem.custom.property=value -password clusterPassword -jcl jclFile [-type jclType] -env environment -port portNumber [-wait true/false] [-printOutput true/false] [-privateKey pathToPrivateKey] [-privateKeyPwd privateKeyPassword]

where:

• jem.custom.property=value could be a set of properties which will be included inside of Job entity and you can use in job listeners or inside of JCL as properties. ONLY the property with name starting with jem.custom. prefix will be managed,otherwise they will be ignored.
• jclFile is the JCL which describes the job (mandatory). Could be a URL or a File path. See here how to use the special JEM URL
• clusterPassword is the Hazelcast password (em.environment.password) needed to connect to the cluster (optional, if not specified, will be asked via console)
• jclType is the type of JCL (optional)
• environment is the name of the execution Environment (alias Hazelcast group name) where to submit the job
• portNumber is the port used by Hazelcast and then by JEM environment
• wait is optional argument and if false the submit process ends immediately, without waiting for the end of the job. Default is true.
• printOutput is optional argument and if true the submit process catches all standard output and error of job execution and displays them. Default is false.
• pathToPrivateKey if login security is enabled is the private user pem key used to connect to the cluster (see Cluster_Security).
• privateKeyPassword if the private user pem key is protected by password

The submitter asks to JEM cluster a unique ID, used as job id and pushes the job in PRE-INPUT queue, waiting for notification of the end of job (if -nowait is not set), sent via Hazelcast publish/subscribe topic features.

To submit a job to JEM remotely, via HTTP protocol, jem_http_submit is used, with the following parameters:

jem_http_submit -Djem.custom.property=value -jcl jclFile [-type jclType] -host nodeURL -user user [-password password] [-wait true/false]

where:

• jem.custom.property=value could be a set of properties which will be included inside of Job entity and you can use in job listeners or inside of JCL as properties. ONLY the property with name starting with jem.custom. prefix will be managed,otherwise they will be ignored.
• jclFile is the JCL which describes the job (mandatory). Could be a URL or a File path. See here how to use the special JEM URL
• jclType is the type of JCL (optional)
• nodeURL is the URL of the web application (mandatory)
• user is the userid needed to authenticate to JEM HTTP interface (mandatory)
• password is the password for user (optional, if not specified, will be asked via console)
• wait is optional argument and if false the submit process ends immediately, polling every 30 seconds via HTTP the JEM node. Default is true.

Note that the command doesn't wait for the end of job execution, it's always executed asynchronously.

Submitting jobs by NodeJS script

All above submits are performed using JAVA classes. Java, for uts nature, starts a JVM and this could use a certain amount of memory.

In some environments, the effort to start a JVM for each job can be very heavy from memory perspective. For this reason, a javascript script has been developed, leveraging on NodeJS, to submit a job (maintaining the same capabilities of the jem_submit script) to reduce the amount of memory.

Before to use it, you must install NodeJS on your machines, used to submit jobs. Here is the download page of NodeJS. Choose the right one for your infrastructure.

NodeJS is enough, no additional packages are necessary.

The jem_submit_js script is used to submit a job (defined by its JCL) leveraging on NodeJS. It takes the following input parameters:

jem_submit_js -password clusterPassword -jcl jclFile [-type jclType] -host nodeURL [-wait true/false] [-printOutput true/false] [-privateKey pathToPrivateKey] [-privateKeyPwd privateKeyPassword] [-localPort number]

where:

• jclFile is the JCL which describes the job (mandatory). This must a FILE path (not a URL).
• clusterPassword is the Hazelcast password (jem.environment.password) needed to connect to the cluster (mandatory)
• jclType is the type of JCL (optional)
• nodeURL is the URL of the web application), used to receive the cluster name and the list of cluster's members (mandatory)
• wait is optional argument and if false the submit process ends immediately, without waiting for the end of the job. Default is true.
• printOutput is optional argument and if true the submit process catches all standard output and error of job execution and displays them. Default is false.
• pathToPrivateKey if login security is enabled is the private user pem key used to connect to the cluster (see Cluster_Security).
• privateKeyPassword if the private user PEM key is protected by password (not used). The private key must be WITHOUT password.
• localPort if port that the script used to stay in listening, waiting for the end of the job. If the port is already used, it increments this value till to find a free port. This is used ONLY if wait is set to true. Default is 7000.

JEM url

JEM provides a URL to submit JCL not reachable from the submitter but available on GFS of JEM itself.

This could be very helpful when you deploy your business logic together with the JCL which should use it. The JCL could be on a folder or in a JAR file. In either cases it's possible to submit a job using that JCL.

The JEM url format is the following:

jem:[gfs-type]:[file-path|jar:file:{JARfile-path}!/{entry}]

where GFS-type could be

1. source if JEM will search the JCL in the GFS source path
2. class if JEM will search the JCL in the GFS class path
3. library if JEM will search the JCL in the GFS library path
4. binary if JEM will search the JCL in the GFS binary path

The file-path is the simple file path where JCL is located inside of GFS path, specified by GFS-type. If the JCL is inside a JAR file, you should specify the JAR file and the full path entry inside the JAR.

Here are a couple of examples:

jem:source:/myjcl/test.xml

JEM will take the test.xml in the folder myjcl inside the source GFS path.

jem:library:jar:file:/myjar/appl.jar!/org/pepstock/jcl/test.xml]

JEM will take the test.xml in the package org.pepstock.jcl inside the jar appl.jar, stored in in the folder myjar, inside the library GFS path.