Week 5: Jenkins for CI and CD Lab Guide

In this lab, we will learn to set up the basic pipeline of CI/CD of Jenkins to integrate it with Git, Github, Maven and Tomcat on two separate EC2 instances.

In this week lab, we will setup this CI/CD pipeline as below:

As you can see, we are dealing with two different EC2 instances (which are very common for microservices).

Here are the overall steps to setup a Jenkin server on the first EC2 instance 1 (Jenkin Server):

• Setup a Linux EC2 Instance 1.

  • Install Java.

  • Install & Setup Maven.

  • Install & Setup Jenkins.

  • Integrate Jenkins with Github and Maven using plugins.

You task:

Let’s setup the EC2 instance for Jenkins Server like so:

• Here are the settings of the Jenkin Server for EC2:

  • Name: Jenkins_Server

  • OS: default Amazon Linux 2023 AMI

• Instance Type: t3.micro

  • Key pair: select and reuse your old key in other labs (devops_project_key)

  • Storage: 15GB (since Jenkins build can be large for big projects)!

  • Security Group:

    • Optional Name: Jenkins_Security_Group

    • Source Type: Anywhere

    • Open two ports:

      • 22 for SSH

      • 8080 for Jenkins

• After launching the instance then, in the EC2 dashboard you should see the Jenkin_Server EC2 running:

• The most important information is the Public IP address of your EC2 instance which will allow us to connect SSH to the server to control it:

• After that, let’s SSH to the Jenkins Server on EC2:

• To keep things simple without dealing with permission, let’s login as a root user and change our current directory to /root:

sudo su -

• Now, you are now a root user with the highest permission privilege to install anything you want. You can check your current directory, which should tell you that you are at root directory:

[You can reuse your existing “Maven_Tomcat_Server” EC2 from last week tutorial to save time instead of setting up a new Tomcat EC2 server from beginning then skip to the “Instructions to setup Deployment on the Jenkins Server” section]

Here are the overall steps to setup a Jenkin server on the first EC2 instance 2 (Tomcat Server):

• Setup a Linux EC2 Instance 2.

  • Install Java.

  • Install & Setup Tomcat.

  • Start Tomcat.

• Access Tomcat Admin Web UI on port 8080.

You task:

Let’s setup the second EC2 instance for Tomcat Server like so:

• Here are the settings of the Tom Server for EC2:

  • Name: Tomcat_Server

  • OS: default Amazon Linux 2023 AMI (Free-tier)

  • Instance Type: t2.micro (Free-tier)

  • Key pair: select and reuse your old key in other labs (devops_project_key)

  • Storage: Default 8 GB is plenty enough!

  • Security Group:

    • Optional Name: Tomcat_Security_Group

    • Source Type: Anywhere

    • Open two ports:

      • 22 for SSH

      • 8080 for Tomcat

      • 80 for HTTP (Optional but useful in the future)

      • 443 for HTTPS (Optional but useful in the future)

• After launching it, you should have two EC2 instances like so:

• After Tomcat installation in the second EC2 instance, Now, let’s go back to the first EC2 instance (Jenkins server) and install deployment plugin for Jenkins to deploy on Tomcat server:

• Install the plugin “Deploy to Container”:

  • Its purpose is specifically for deploying Java web applications (e.g., .war files) to servlet containers like Tomcat.

• Let’s add credentials of Tomcat users to deploy our web application:

• Choose “System” in the credentials context:

• Choose “Global” which are credentials that available to all jobs, agents, and users across the entire Jenkins instance:

• Remember that we have an admin user of Tomcat to have a “manager-script” role so this specific “admin” user can run script from one instance to manage another instance (from Jenkin Server to Tomcat Server):

• Now, you should have the credentials like so:

• Nice, now we are ready to create a new Jenkins job to build and deploy on Tomcat Server:

• Please fill in the details of this job:

  • Description:

• Git:

• Maven Build:

• Post-build Actions (Things happens after building completion):

• We are done with the configurations of that Jenkins job:

• It’s time to build and deploy with our new Jenkins job:

• Let’s look at the build log and try to understand it:

• Alright, it seems alright! Git pull -> Maven Build -> Deploy on Tomcat done! Let’s check the website:

Super glorious! We love to see that!

• You can login as Tomcat admin to check out the Manager App to see if your War file is in the webapps directory:

• That’s pretty awesome!

• Poll SCM is very easy to set up as it does not require additional integration or setup outside Jenkins. However, it is quite inefficient as Jenkins continuously checks if there is any new change of the repository, even when no changes are made. As a result, this consumes more resources, especially when polling frequently or with large repositories.

• GitHub Hook Trigger for GITScm Polling is more efficient as there is no periodic polling; Jenkins acts only when notified of changes (real-time triggers). As a result, it reduces unnecessary load on both Jenkins and Git servers. However, it has more complex steps to set up as it requires GitHub repository admin rights to set up webhooks.

Instead of using Poll SCM, can you try to use the “GitHub hook trigger for GITScm polling” option to trigger builds efficiently?

Here are some step by step guidance to set it up:

• Configure GitHub Repository:

  • Go to your repository on GitHub.

  • Navigate to Settings > Webhooks.

  • Click on Add webhook.

    • Payload URL : Use http://<JENKINS_URL>:<PORT>/github-webhook/ (replace <JENKINS_URL> and <PORT> with your Jenkins URL and port).

    • Content type : Select application/json.

    • Secret : Leave blank (optional).

    • SSL verification : Enable.

    • Trigger events : Select Just the push event.

• Install Required Jenkins Plugins

  • GitHub Plugin (if not yet installed)

• Enable GitHub Hook Trigger in Jenkins Job

  • Select GitHub hook trigger for GITScm polling for the Jenkins job!

• Test the Webhook

  • Go back to your GitHub repository settings.

  • Under Webhooks, click on your newly created webhook.

  • Click Redeliver to send a test payload to Jenkins.

• Go to Jenkins Dashboard > Manage Jenkins > System Log.

• Verify that the webhook payload is received by Jenkins.

• Test with a real Commit

  • Go to the Github repo

  • Make a change and commit

  • Check your Jenkins job to see if the build is triggered automatically.

  • In the log of the build triggered by Github Webhook, you should see this:

Objective: In the tutorial so far, to keep it simple, you have been working with a public Github repo. However, in real-life projects, most of the company projects are in private repos.

When working with private GitHub repositories, Jenkins needs proper authentication credentials to access the repository. This challenge guides you through setting up Jenkins to securely pull code from a private repository.

󰠁 Before You Start: Create Your Own Private GitHub Classroom Repo

If you don’t already have a private GitHub repository, follow the instructions in the Github Classroom Guide on your course page to create one. This is also good practice for your upcoming in-class test.

⚠ Reminder: Your GitHub repository must be private to simulate real-world CI/CD scenarios and prepare you for the exam.

📦 After receiving your private GitHub Classroom repo:

• Add some files to your repo (e.g., a simple README or .java files), or upload a sample Maven Web Project.

• This will allow you to verify later that Jenkins can clone or pull the repository contents successfully when configured.

🔧 Step-by-Step Instructions:

• Step 1: Generate classic Personal Access Token (PAT) from GitHub

  • Select scopes:

    • repo ( Full control of private repositories)

    • admin:repo_hook (To read repo webhooks)

    • deplete_repo (Delete repositories)

• Step 2: Add GitHub Credentials to Jenkins

  • Manage Jenkins > Credentials > System

  • The credentials (e.g., Global credentials), click Add credentials.

    • Kind: Username with password.

    • Username: Your GitHub username.

    • Password: Paste the generated PAT here.

    • ID: Enter a recognizable ID like github-private-repo-credentials.

    • Description: Add an optional description (e.g., "GitHub PAT for private repositories").

• Step 3 : Configure Jenkins Job to Use the Credentials

  • Under Source Code Management, select Git.

  • Click on Credentials and select the credential ID you created (github-private-repo-credentials).

• Step 4 : Test the Configuration

  • Build the Jenkins job.

  • Check the build log:

    • If successful, you should see Jenkins fetching the repository code.

    • If there’s an error, ensure the PAT has the correct permissions and was copied correctly.

Notes:

• For security, never expose your Personal Access Token in job logs or configuration files.

• If your team uses SSH for GitHub authentication, you can configure Jenkins with an SSH private key instead (you should research how to setup SSH private key instead of PAT for more secure Github authentication)

Objective : In a professional environment, SSH key pairs are often used to securely authenticate Jenkins with external services like GitHub. Your task is to set up SSH key-based authentication for Jenkins and use it to securely pull code from a private GitHub repository.

Here are some general steps for your hints:

• Generate SSH Key Pair for Jenkins

  • SSH into the Jenkins server (EC2 Instance 1).

  • Generate an SSH key pair for Jenkins using the following command:

    • ssh-keygen -t rsa -b 4096 -C "your-email@your-company.com"

  • Press Enter to save the key to the default location (~/.ssh/id_rsa) and skip the passphrase by pressing Enter again.

  • Pay attention to the location of these public and private key files:

    • Private Key: ~/.ssh/id_rsa

    • Public Key: ~/.ssh/id_rsa.pub

• Add the Public Key to GitHub

  • Copy the public key content:

    • cat ~/.ssh/id_rsa.pub

  • Go to GitHub:

    • Navigate to Settings > SSH and GPG Keys.

    • Click New SSH Key.

    • Paste the copied key content and save it.

• Add the Private Key to Jenkins

  • Access the Jenkins dashboard and navigate to: Manage Jenkins > Manage Credentials > (Global) > Add Credentials.

  • Select the credential type SSH Username with Private Key and configure it:

• Username: git

  • Private Key: Select Enter directly and paste the private key content:

    • cat ~/.ssh/id_rsa

    • ID: Use an easily recognizable ID, such as ssh-key-github.

    • Description: Add a description like SSH Key for GitHub.

• Update Jenkins Git Host Key Verification Strategy

  • Jenkins is enforcing strict host key verification, and you need to configure it appropriately to make it work for the purpose of testing!

  • Go to Manage Jenkins > Security

• Choose No host key verification temporarily for testing (not recommended for production environments).

• Configure Jenkins Job with SSH-based GitHub Repository URL

  • Go to your Jenkins job settings.

  • Under Source Code Management, select Git.

  • Enter the repository URL in SSH format:

    • git@github.com:username/private-repo.git

  • In the Credentials field, select the SSH credentials (ssh-key-github) you added earlier.

• Test the Configuration

  • Save the Jenkins job configuration.

  • Trigger a build for the job.

  • Verify in the build logs that Jenkins successfully clones the private repository.

• Troubleshooting Tips (if something goes wrong)

  • If the build fails with permission errors:

    • Ensure the SSH key is correctly added to your GitHub account.

    • Verify that the GitHub repository URL uses the git@github.com format.

  • Test the connection from the Jenkins server to GitHub:

    • ssh -T git@github.com

    • You should see a success message indicating a successful authentication.

[Question] Why is SSH key-based authentication preferred over Personal Access Tokens (PATs) for Jenkins in CI/CD pipelines?

Objective : Ensure that the CI/CD pipeline halts the deployment process if a critical test (such as verifying the presence of an HTML component) fails. This challenge demonstrates how to add automated validation as a gatekeeper in your Jenkins pipeline and prevent faulty code from being deployed to production.

Problem Statement: In a typical CI/CD pipeline, automated testing ensures code quality before deployment. If a test fails, the deployment process should stop to prevent incomplete or faulty features from reaching the server. Your task is to:

• Integrate a test step into the Jenkins pipeline to validate the presence of an expected HTML component in the index.html file (e.g., <h1>DevOps</h1>).

• Stop the deployment to a Tomcat server if the test fails.

• Notify the development team when the deployment is blocked due to a failed test.

Steps to Complete the Challenge

• Modify your existing Jenkins job to include a testing stage before the deployment stage:

  • Add a Test Step: Build Step > Execute Shell.

  • Enter the following script to test if an HTML component exists:

    • This script checks for a title element in the index.html file. Adjust the search term in the grep command as needed.

    • exit 1 ensures the build step fails if the HTML component is not found or the index.jsp file is missing. Jenkins interprets any non-zero exit code as a failure, which marks the build as FAILED, which will prevent subsequent stages like deployment.

# Navigate to the cloned repository's location
cd $WORKSPACE
# Check if index.html exists
if [ ! -f src/main/webapp/index.jsp ]; then
  echo "Error: index.jsp not found!"
  exit 1
fi

# Search for a specific HTML component
if grep -q "<title>Hello DevOps Students!</title>" src/main/webapp/index.jsp; then
  echo "HTML component found!"
else
  echo "Error: HTML component not found!"
  exit 1
fi

• [Optional] Configure a notification system (e.g., email or Slack) to alert the team about the failure. Use the Email Notification Plugin or Slack Plugin for this purpose.

• Test the Pipeline:

  • Modify the index.jsp file in your Github repository to intentionally miss the required HTML component.

  • Commit and push the changes to your GitHub repository.

  • Trigger the pipeline and verify:

    • The pipeline stops at the testing stage with a failure.

    • Notifications are sent to the team about the failure.

  • Restore the correct index.jsp file, commit, and push again. Ensure the deployment proceeds successfully.

Success build log screenshot:

Fail build log screenshot:

This is a naive example of testing, intended purely for learning purposes. While the simple grep command demonstrates how to implement basic validations in Jenkins, it is not sufficient for real-world web application testing. For robust and scalable testing in real-world applications, consider the following:

• Selenium : For browser-based automation and end-to-end testing.

• JUnit/TestNG : For unit and integration testing in Java applications.

• Mockito : For mocking dependencies during Java tests.

• Spring Boot Test Tools : For REST API and service layer testing.

• JaCoCo : For code coverage analysis.

• Cypress/Playwright : Modern alternatives to Selenium for end-to-end testing.

• Jenkins Plugins : Use plugins like JUnit, Selenium, and JaCoCo to integrate test results into your pipeline.

These tools enhance testing for web applications, ensuring reliability and production-readiness.

Extra Reading:

• Setup CI/CD Pipeline with Jenkins by using AWS CodeBuild and AWS CodeDeploy