Run Configurations

Execute Jobs / Transformations on specific nodes or in a Pentaho Cluster ..

Pentaho Data Integration provides advanced clustering and partitioning capabilities that allow organizations to scale out their data integration deployments.

In this guided demonstration, you will:

• Configure Master & Slave Nodes

• Execute RUN Configurations

tr_hello_world.ktr

1. Master & Slave nodes

So lets start scaling out by adding some servers (nodes).

These can be defined as either:

• Master: node is responsible for distributing work among the worker nodes and ensuring high availability and scalability of the system.

• Slave (Worker): node in Pentaho is an instance that can execute Pentaho work items, such as PDI jobs and transformations, with parallel processing, dynamic-scalability, load-balancing, and dependency-management in a clustered environment

Master Node

You can indiviually start the carte instances or execute the following command to deploy all 3 at the same time .

cd
cd ~/Scripts
./start_carte.sh

1. In a terminal execute the following command.

cd
cd ~/Pentaho/design-tools/data-integration
sh carte.sh localhost 12000

Master node - port 12000

Slave Nodes

1. In a new terminal execute the following command (Slave A).

cd
cd ~/Pentaho/design-tools/data-integration
sh carte.sh localhost 12100

Slave node A - port 12100

2. In a new terminal execute the following command (Slave B).

cd
cd ~/Pentaho/design-tools/data-integration
sh carte.sh localhost 12200

Slave node B - port 12200

You should now have 3 terminals, each running a Carte instance.

Please dont close the terminals ..!

2. Configure Nodes

1. Open the tr_hello_world transformation.

2. Select the View tab

3. Highlight the Slave server option; right mouse click and select: New

Configure Nodes

Option	Description
Server name	The name of the slave server.
Hostname or IP address	The address of the device to be used as a slave.
Port (empty is port 80)	Defines the port you are for communicating with the remote server. If you leave the port blank, 80 is used.
Web App Name (required for Pentaho Server)	Leave this blank if you are setting up a Carte server. This field is used for connecting to the Pentaho server.
User name	Enter the user name for accessing the remote server.
Password	Enter the password for accessing the remote server. (cluster/cluster)
Is the master	Enables this server as the master server in any clustered executions of the transformation.

Below are the proxy tab options:

Option	Description
Proxy server hostname	Sets the host name for the proxy server you are using.
The proxy server port	Sets the port number used for communicating with the proxy.
Ignore proxy for hosts: regexp | separated	Specify the server(s) for which the proxy should not be active. This option supports specifying multiple servers using regular expressions. You can also add multiple servers and expressions separated by the ' | ' character.

Master Node

1. Enter the following settings to configure the Master node:

Master Node

Slave Nodes

1. Enter the following settings to configure the Slave node A:

Slave Node A

2. Enter the following settings to configure the Slave node B:

Slave Node B

Ok .. now we're ready to RUN transformations on specific nodes.

3. RUN Configurations

Now we have our 3 nodes up and running, lets configure some RUN configurations to execute our Transformations on specific nodes.

Some ETL activities are lightweight, such as loading in a small text file to write out to a database or filtering a few rows to trim down your results. For these activities, you can run your transformation locally using the default Pentaho engine.

Some ETL activities are more demanding, containing many steps calling other steps or a network of transformation modules. For these activities, you can set up a separate Pentaho Server dedicated for running transformations using the Pentaho engine.

Other ETL activities involve large amounts of data on network clusters requiring greater scalability and reduced execution times. For these activities, you can run your transformation using the Spark engine in a Hadoop cluster.

tr_hello_world

3.1 Nodes

Pentaho local is the default run configuration. It runs transformations with the Pentaho engine on your local machine. You cannot edit this default configuration.

1. Ensure you have configured the Nodes.

Slave Nodes

2. To create a new run configuration, right-click on 'Run configurations' folder and select New.

Run configuration

3. Enter the following configuration details, ensuring that you select the Pentaho (KETTLE) engine.

Master - RUN configuration

3. When you come to RUN the transformation, select Master node.

RUN configuration - Master

As you can see from the Results:

• Transformation is executed on the Master Node

• As Monitor tab displays the Step Metrics

RUN configuration - Master

3. Take a look at the Master Terminal.

Master Terminal

Give it a go with other RUN configurations .. Just Slave A / B

3.2 Cluster schema

A cluster schema is essentially a collection of slave servers. In each schema, you need to pick at least one slave server that we will call the Master slave server or master.

The master is also just a carte instance but it takes care of all sort of management tasks across the cluster schema. In the Spoon GUI, you can enter this metadata as well once you started a couple of slave servers.

The workflow in a clustered Pentaho transformation is as follows:

• The job entry or the transformation connects to the cluster master node, which is responsible for coordinating the execution of the transformation steps on the cluster slave nodes.

• The master node sends the transformation metadata and the cluster schema to the slave nodes, and assigns each step to one or more nodes based on the cluster schema.

• The slave nodes execute the assigned steps and exchange data with each other using sockets or shared files, depending on the partitioning method and the clustering plugin used.

• The master node monitors the progress and status of the slave nodes, and collects logging information and performance metrics from them.

• The master node reports the outcome of the transformation execution to the job entry or the transformation that initiated it.

Cluster Schema

1. To create a new run configuration, right-click on 'Kettle cluster schemas' folder and select New.

Kettle cluster schema

2. Enter the following configuration details.

Pentaho Cluster schema

Option	Description
Schema name	The name of the clustering schema
Port	Specify the port from which to start numbering ports for the slave servers. Each additional clustered step executing on a slave server will consume an additional port. Note: To avoid networking problems, make sure no other networking protocols are in the same range.
Sockets buffer size	The internal buffer size to use
Sockets flush interval rows	The number of rows after which the internal buffer is sent completely over the network and emptied.
Sockets data compressed?	When enabled, all data is compressed using the Gzip compression algorithm to minimize network traffic
Dynamic cluster	If checked, a master Carte server will perform failover operations, and you must define the master as a slave server in the field below. If unchecked, Spoon will act as the master server, and you must define the available Carte slaves in the field below.
Slave Servers	A list of the servers to be used in the cluster. You must have one master server and any number of slave servers. To add servers to the cluster, click Select slave servers to select from the list of available slave servers.

3. To create a new run configuration, right-click on 'Run configurations' folder and select New.

4. Enter the following configuration details, ensuring that you select the Pentaho (KETTLE) engine.

Clustered - RUN configuration

You can set for the logs to be created locally.

The 'Show transformations' option is useful for documentation, as diagrams illustrating the dataflow are created.

3.3 Clustered

A Carte cluster consists of two or more Carte slave servers and a Carte master server. A Carte server is a lightweight web service that can execute jobs and transformations remotely. A Carte cluster can speed up the processing of transformations by distributing the work across multiple Carte slave nodes, while the Carte master node tracks the progress. A Carte cluster can be static or dynamic. A static Carte cluster has a fixed schema that specifies the nodes in advance. A dynamic Carte cluster allows you to add or remove nodes at runtime.

Pentaho can also connect to other types of clusters, such as Hadoop clusters, to leverage big data processing capabilities. For example, Pentaho can connect to secured and unsecured MapR clusters, which are Hadoop distributions that provide high performance, reliability, and security. To connect to a MapR cluster, you need to configure the cluster, install any required services and service client tools, and test the cluster

1. Highlight the Hello World step, right mouse click and select the option Clusters from the drop down menu.

Clusters

4. Select 'Pentaho' cluster schema.

Pentaho - Cluster schema

Notice that the step will indicate the number of Slave nodes it will be executed on.

Indicated nmber of Slave Nodes

5. RUN the transformation with Clustered configuration.

RUN configuration - Clustered

A bunch of Tabs will appear for each node which display the Metrics and Metadata transformations.

6. Take a look at the Tabs (example below is for Slave A).

Metadat transformation - Slave A

Here you can see that the records are streamed from Slave node A back to the Master node.

Metrics - Slave A

The metrics indicate that Slave Node A ingested the records.

Logs - Slave A

The Logs indicate Slave A successfully read the transformation metadata dispatched from the Master node and executed the step, streaming the resulting dataset back to the Master node.