Jenkins

Used as an open-source scheduler ..

Jenkins is an open-source automation server that enables developers to build, test, and deploy software efficiently. It's mainly used for Continuous Integration (CI) and Continuous Delivery (CD), automating the parts of software development related to building, testing, and deploying, facilitating continuous improvement and collaboration.

CI/CD

Setup

The following section is for Reference only.

Jenkins + plugins have been installed and configured.

Docker

Docker is a platform designed to help developers build, share, and run container applications.

Docker Compose is a tool for defining and running multi-container applications.

1. Run update & upgrade (optional).

sudo apt update -y && sudo apt upgrade -y

1. Create a Jenkins folder.

cd
mkdir ~/Jenkins

1. Create docker-compose.yml

cd
cd ~/Jenkins
nano docker-compose.yml

services:
  jenkins:
    image: jenkins/jenkins:lts
    restart: always
    privileged: true
    user: root
    ports:
      - 9080:8080
      - 50000:50000
    container_name: jenkins-1
    volumes:
      - /home/pentaho/Jenkins/jenkins_configuration:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock

image          — lts will download latest version.
ports          — defines ports: 9080 (www) and 50000 (api)
container_name — the name of our container: jenkins-1
volumes        — these are the virtual volumes used by the container. 

1. Save.

CTRL + O
Enter
CTRL + X

1. Create jenkins container.

cd
cd ~/Jenkins
sudo docker compose up -d

[+] Running 13/13
 ✔ jenkins Pulled                                                         10.3s 
   ✔ ca4e5d672725 Pull complete                                            4.1s 
   ✔ 5c2c33065c08 Pull complete                                            6.4s 
   ✔ 92488061783e Pull complete                                            6.6s 
   ✔ 283fc5f22098 Pull complete                                            6.6s 
   ✔ a9c4c37656d4 Pull complete                                            6.7s 
   ✔ ba8c648ced13 Pull complete                                            7.5s 
   ✔ 1cf1dfaee3c9 Pull complete                                            7.5s 
   ✔ 94f0d5472c4d Pull complete                                            7.6s 
   ✔ f8b0d06461dd Pull complete                                            8.5s 
   ✔ 43dde4536f7b Pull complete                                            8.6s 
   ✔ cdaf12c3ce9b Pull complete                                            8.6s 
   ✔ 70d4bec61dab Pull complete                                            8.6s 
[+] Running 2/2
 ✔ Network jenkins_default  Created                                        0.2s 
 ✔ Container jenkins-1        Started                                        3.6s 

1. Verify the logs

sudo docker compose logs --follow

jenkins  | 2024-08-14 09:49:09.249+0000 [id=47]	INFO	jenkins.install.SetupWizard#init: 
jenkins  | 
jenkins  | *************************************************************
jenkins  | *************************************************************
jenkins  | *************************************************************
jenkins  | 
jenkins  | Jenkins initial setup is required. An admin user has been created and a password generated.
jenkins  | Please use the following password to proceed to installation:
jenkins  | 
jenkins  | 3d56d7ddc87c459ebaafe1117f7f56e7
jenkins  | 
jenkins  | This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
jenkins  | 
jenkins  | *************************************************************
jenkins  | *************************************************************
jenkins  | *************************************************************

1. Make a note of the password: 3d56d7ddc87c459ebaafe1117f7f56e7

2. You can also retrieve the password with:

cd
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

---

Configure Jenkins

1. Access Jenkins UI.

http://localhost:9080localhost

Link to Jenkins UI

1. Copy and paste the password.

Setup Admin account

1. Click 'Continue' & keep the default option 'Install suggested plugins'.

Install suggested plugins

Default plugins

1. Create Admin User.

Username	admin
Password	Welcome123!
Full name	system admin

admin user

1. Save & Finish. You are now able to access Jenkins with the default admin account.

Jenkins URL confirmtion

Jenkins Agent

This section is for Reference only.

Best Practice is to configure a specific system account - Jenkins

The Jenkins architecture is designed for distributed build environments. It allows us to use different environments for each build project balancing the workload among multiple agents running jobs in parallel.

The Jenkins controller is the original node in the Jenkins installation. The Jenkins controller administers the Jenkins agents and orchestrates their work, including scheduling jobs on agents and monitoring agents. Agents may be connected to the Jenkins controller using either local or cloud computers.

The agents require a Java installation and a network connection to the Jenkins controller.

1. Generate an SSH key. This key will allow the controller to access the agent via SSH.

cd
ssh-keygen -f ~/.ssh/jenkins_agent

Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/pentaho/.ssh/jenkins_agent
Your public key has been saved in /home/pentaho/.ssh/jenkins_agent.pub
The key fingerprint is:
SHA256:w+logzz6LDa6zZqmBNqbjLBAmF8Tw06vIuBwYFReA1U pentaho@pentaho
The key's randomart image is:
+---[RSA 3072]----+
| ..oo+.E         |
|. ... .          |
|.. .=            |
|oo o + . .       |
|*.. + . S        |
|B+ o + o .       |
|=o+ = + .        |
|+X+* o .         |
|@=B+o            |
+----[SHA256]-----+

This command creates two files:

jenkins_agent - private key.

jenkins-agent.pub - public key.

1. Navigate to your Jenkins dashboard.

http://localhost:9080/localhost

Link to jenkins UI

1. Log in as user: Jenkins

Jenkins UI

1. Go to Manage Jenkins option in main menu and click on Credentials button.

Jenkins UI - Credentials

1. From the drop down select: Add Credentials

Add credentials to

1. Fill in the form:

Configure connection to Agent

Kind	SSH Username with private key
id	jenkins
Description	jenkins ssh key
Username	jenkins - jenkins' user already exists by default in the jenkins_agent-1 container.
Private Key	select Enter directly and press the Add button to insert the content of your private key file at ~/.ssh/jenkins_agent
Paraphrase	fill your passphrase used to generate the SSH key pair.

1. Click on 'Create'.

Credentials to access agents - SSH Key

---

Jenkins Agent on Docker

1. Run the command to start your first agent.

docker run -d --rm --name=jenkins_agent-1 -p 22:22 \
-e "JENKINS_AGENT_SSH_PUBKEY=[your-public-key]" \
jenkins/ssh-agent:alpine-jdk17

• Replace the tag [your-public-key] for your own SSH public key.

• Your public key value in this example could be found by issuing : cat ~/.ssh/jenkins_agent_key.pub on the machine your created it. Do not add the square brackets [] around the key value

• The value of [your-public-key] MUST include the full contents of your .pub file, including the ssh-XXXX prefix.

• Ex: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAQQCo9+BpMRYQ/dL3DS2CyJxRF+j6ctbT3/Qp84+KeFhnii7NT7fELilKUSnxS30WAvQCCo2yU1orfgqr41mM70MB

• If your machine already has a ssh server running on the 22 port (if you logged onto this machine thanks to the ssh command, that’s the case), you should use another port for the docker command, such as -p 4444:22

or you can modify the docker-compose.yml to include the agent and public key.

1. Bring the jenkins-1 container down.

cd
cd ~/Jenkins
docker-compose down

1. Edit the docker-compose.yml

cd
cd ~/Jenkins
nano docker-compose.yml

Add the ~/.ssh/jenkins_agent.pub - the public key that will authentiacte the connection to the agent.

services:
  jenkins:
    image: jenkins/jenkins:lts
    restart: always
    privileged: true
    user: root
    ports:
      - 9080:8080
      - 50000:50000
    container_name: jenkins-1
    volumes:
      - /home/pentaho/Jenkins/jenkins_configuration:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
  agent:
    image: jenkins/ssh-agent:jdk17
    privileged: true
    user: root
    container_name: jenkins_agent-1
    expose:
      - 22
    environment:
      - JENKINS_AGENT_SSH_PUBKEY=ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABgQCfQR3NKeoP+OkRbXqDmltwfxq6vYYrjFP7+kiKHrqB/xIuHbrcWGrRpb65qaPXstzxZLopiN1lkdn0kCeJAHwe38tHB+CJESqjvGemq0hRPAY/YGT1DqyP0l5IQAck6NGRC6VM7Gto/vGTgJQF5eNh+hPa+4wfTvkzfwy5gB5rJN9zfqW08uYNurqOJCHoCBb0FnyRVCO3WzWReib+A5YVP880jOLLR3jhyNkJEjsDH1yFPPc5sS11bHcK9yl8NZeVsi9KpPFCOYSuZCmBai+uXXX+vMTwQ95k4qUMorQqV8ctZ1U8b7PlcuBlv0OSz+LyxOBELPxHYuoRGn4T5bWiQCUakSzaTmsstsyRFQXL5tG+TZwGxzvsbyGNUY7ybvh08tr4a8XowkctrdLEJKKr3n+Meag1FpHwEI0c+X96hL3RyC6+pAOTeW1ZbljydNTW59LM47nzrqmGn/mZF6PHSlmjfy9IUWZuHT1E0pTfq+ro+XNfNHcnhnqzIu7v5OM= pentaho@pentaho

[+] Running 9/9
 ✔ agent Pulled                                                            6.6s 
   ✔ ca4e5d672725 Already exists                                           0.0s 
   ✔ e5f960337628 Pull complete                                            0.4s 
   ✔ a05e468af172 Pull complete                                            4.0s 
   ✔ e60e3fc02759 Pull complete                                            4.0s 
   ✔ 4f4fb700ef54 Pull complete                                            4.0s 
   ✔ dee10b008112 Pull complete                                            4.9s 
   ✔ 670563ebccf3 Pull complete                                            4.9s 
   ✔ 1b4075dda320 Pull complete                                            5.0s 
[+] Running 3/3
 ✔ Network jenkins_default    Created                                      0.1s 
 ✔ Container jenkins          Started                                      1.5s 
 ✔ Container jenkins_agent-1  Started                                      1.5s 

1. Check that the Jenkins Docker Agent is up and running.

Portainer - jenkins containers

---

Configure the Agent

This is a very basic setup - everything is installed on a single server, i.e. from the master node we can deploy a new slave node & run our Jobs using the deployed Agent.

1. Navigate to your Jenkins dashboard.

2. Log in as user: Jenkins

Jenkins UI

1. Go to Manage Jenkins option in main menu and click on Nodes button.

Or click on the Set Up agent message.

Configure Node

1. Click on New Node button.

Create New Node

1. Enter the following details:

New Node

x

Node1 configuration

Jenkins

Let's run through a couple of Jenkins scenarios:

Periodic Build - demonstrates the first stages of the CI/CD pipeline. Developers commit to a Github repository which, periodically, triggers a 'build'.

The 'build' executes a shell 'PUT' script which uploads the Jobs & Transformations into the Pentaho Repository.

Commit Build - a Github commit triggers a 'build'.

Periodic Build

You could simply set a CRON schedule to trigger the 'build' process ..

1. Log into Jenkins.

http://localhost:9080localhost

Link to Jenkins

Username	admin
Password	Welcome123!

1. Click on 'Create a Job' or 'New Item'.

2. Provide a name - Periodic Build - for the item and then choose: 'Freestyle project' & click: 'OK'

Freestyle project - Periodic Build

1. Set the following project options - execute every 5 mins:

Section	Option	Value
Source Management
Git	Repository URL	https://github.com/jporeilly/Jenkins-Github
Branches to build
Branch Specifier		**
Build Triggers
Build periodically	Schedule	H/5 * * * *
Build Steps
Execute shell	Command	curl -H "Content-Type: application/xml" -d @Job1.kjb -X PUT -u admin:password -i http://pentaho.pentaho.lab:8080/pentaho/api/repo/files/:public:Jenkins:Job1.kjb
		curl -H "Content-Type: application/xml" -d @Transformation1.ktr -X PUT -u admin:password -i http://pentaho.pentaho.lab:8080/pentaho/api/repo/files/:public:Jenkins:Transformation1.ktr
		curl -H "Content-Type: application/xml" -d @Transformation2.ktr -X PUT -u admin:password -i http://pentaho.pentaho.lab:8080/pentaho/api/repo/files/:public:Jenkins:Transformation2.ktr

After specifying the Repository URL, and the error "Failed to connect to repository..." is displayed - indicates Git has not been installed.

If the Repository is private, Jenkins wil validate the Github credentials when pulling the source code.

Source Code Management

Configure Git Repository

Build Triggers

Branches to build

Build periodically - every 5 minutes

Build Steps

Build Steps

Before you click Save, just check:

• Pentaho Server is up and running.

• Create a Public/Jenkins folder & is empty.

cd
cd/opt/pentaho/server/pentaho-server/
sudo ./start-pentaho.sh

1. Check the 'Build Status' in the Dashboard.

1. Click on a build

1. View the 'Console Ouput'.

Console Output

1. Finally .. check the Pentaho Repository.

Pentaho Repsoitory

Commit Build

In CI/CD we want to trigger the build once we commit our code to git.

There are at least two approaches to achieving this:

You could have Jenkins continuously polling the git repository for changes or more efficiently, have Github tell Jenkins about any commits that happen.

1. Log into Jenkins.

http://localhost:9080localhost

Link to Jenkins

Username	admin
Password	Welcome123!

1. Click on 'New Item'.

New Item

1. Provide a name - Commit Build - for the item and then choose: 'Freestyle project' & click: 'OK'

Freestyle Project - Commit Build

1. Set the following project options - Git commit:

Section	Option	Value
Source Management
Git	Repository URL	https://github.com/jporeilly/Jenkins-Github
Branches to build
Branch Specifier		**
Build Triggers
Build periodically	Schedule	H/5 * * * *
Build
Execute shell	Command	curl -H "Content-Type: application/xml" -d @Job1.kjb -X PUT -u admin:password -i http://pentaho.pentaho.lab:8080/pentaho/api/repo/files/:public:Jenkins:Job1.kjb
		curl -H "Content-Type: application/xml" -d @Transformation1.ktr -X PUT -u admin:password -i http://pentaho.pentaho.lab:8080/pentaho/api/repo/files/:public:Jenkins:Transformation1.ktr
		curl -H "Content-Type: application/xml" -d @Transformation2.ktr -X PUT -u admin:password -i http://pentaho.pentaho.lab:8080/pentaho/api/repo/files/:public:Jenkins:Transformation2.ktr

After specifying the Repository URL, and the error "Failed to connect to repository..." is displayed - indicates Git has not been installed.

If the Repository is private, Jenkins wil validate the Github credentials when pulling the source code.

x

x