The Dev World – Sergio Lema

In this article I create my first CI/CD Pipeline with GitlabCI. I explain the main concepts of GitlabCI, jobs, stages and conditions. And I show how to configure a runner.

Content

• Gitlab-CI
• Jobs
• Stages
• Conditions
• Runners
• YAML file
• Hands-On
• Specific Runner Configuration
• More Gitlab-CI Keywords

All the details can be found in the explanatory video.

The code used in this article is available in this repository.

Gitlab-CI

Gitlab is known to be a remote GIT server where everyone can store its project. It’s like GitHub, and as with the GitHub Actions, I have the CI CD section, a continuous integration and continuous deployment section in Gitlab called GitlabCI.

Jobs

The CICD pipelines in GitLab are first composed by jobs. A job is the action I want to execute in my project: compile, run the unit tests, run the integration tests, deploy somewhere, and more. The jobs can have a single command or multiples. If a single command fails, the job will stop and return an error to the pipeline.

Stages

Then comes the stages. A stage is a group of jobs. I may have a stage to test and independent jobs to run the unit tests, the integration tests and the load tests. All the jobs in a stage can run in parallel. When all the jobs finishes successfully, the next stage is started. For example, the packaging stage or the deploy stage that can be executed only if the tests are successful.

Conditions

And finally comes the conditions. I can apply some conditions to the jobs, as dependency between them. This makes my jobs to run sequentially. Or I can execute some jobs only in a specific branch or tag. This way, I can have the master branch which only runs the unit test and the deploy, and the other branches which run the integration tests but not the deploy.

Runners

But where are executed those jobs? In Runners. With GitlabCI, I can configure multiple Runners to accept jobs. I can have shared Runners or dedicated Runners. The shared Runners will be used for all the projects in my GitHub CI instance. The specific Runners will be dedicated to a single project.

In the current article, I will create a workflow example where I will use a specific Runner from my machine. When a new commit arrives to Gitlab, GitlabCI will check the conditions to run the job and if some Runner is available it will assign the job.

YAML File

The configuration of my pipeline will be done in a YAML file. This YAML file will be located inside the project. This way, the project contains the way it must be built and deployed all together.

Hands-On

stages:
  - test
  - deploy

Let’s create a simple example. I first need to define the two stages I want: the test stage and the deploy stage. The test stage will contain the unit test for the master branch and the integration test for the other branches. And the deploy stage will only be available for the master branch. As I have nowhere to deploy my application yet, this will be a fake deploy.

test-all:
  stage: test
  except:
    - main
  script:
    - mvn clean install

test-all is the name of my job. I can name it as I want. test is the stage where my job is located. With the except keyword, I indicate that I don’t want to execute this job on main. And finally, with the script keyword are the commands to be run by this job.

test-unit:
  stage: test
  only:
    - main
  script:
    - mvn clean package
    - cp target/backend-social-network*.jar backend-social-network-${CI_COMMIT_SHA}.jar
  artifacts:
    name: backend-social-network
    expire_in: 1 week
    paths:
      - backend-social-network-${CI_COMMIT_SHA}.jar

As before, in the first line is declared the name of the job, test-unit. Then the associated stage. This time, I indicate that this job will only be executed in the main branch. Finally, the script to run the unit tests. I’ve added other another command which copied the JAR file and rename it with the hash of the commit. As this job will be executed only in the main branch, I want to keep a trace of all that will be built and deployed.

CI_COMMIT_SHA is the hash of the commit which triggered the pipeline. It’s a Gitlab environment variable. There are hundreds more, as the local branch, the project name, the pipeline ID, and more. I can also set my own environment variables per job or per pipeline.

variables:
  MY_VARIABLE: its-value
  ANOTHER_VARIABLE: 123

In the last part of the job, from line 8 and on, I save back the JAR file as an artifact. This artifact will be available in the Gitlab CI interface. We will see it later. Let’s go now to the latest job, the deploy.

deploy:
  stage: deploy
  needs: [test-unit]
  only:
    - main
  script:
    - ls backend-social-network*.jar

Here, I’ve introduced a new concept: the needs. The dependency with the test-unit job. This means that the deploy job will only be executed if the test-unit job is successful. That’s all for the pipeline.

Specific Runner Configuration

I will create one on my local machine, attach it to my Gitlab account and use it to execute the pipelines. In the settings section of my project at Gitlab, I have the CI CD tab. In the runners section, I can see the shared Runners and the specific Runners.

To use the specific Runners, I must ensure that the shared Runners are disabled. Then, click on Show runner installation instructions to create my own Runner.

I will create my Runner not in my machine but in a Docker image. I will start a Java image and run all of those commands inside.

docker run -it openjdk:11.0.16-jdk bash
>
> curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
> chmod +x /usr/local/bin/gitlab-runner
> useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
> gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
> gitlab-runner start
>
> gitlab-runner register --url https://gitlab.com/ --registration-token $REGISTRATION_TOKEN
Runtime platform 	arch=amd64 os=linux pid=45 revision=bbcb5aba version=15.3.0
Running in system-mode

Enter the Gitlab instance URL (for example, https://gitlab.com/)
[https://gitlab.com/]:
Enter the registration token:
[xxx]:
Enter a description for the runner:
[9141e579f65f]: Java runner
Enter tags for the runner (comma-separated):

Enter optional maintenance note for the runner:

Registering runner... succeeded
Enter an executor: docker+machine, custom, docker, parallels, ssh, virtualbox, docker-ssh, shell, docker-ssh+machine, kubernetes:
shell
Runner registered successfully. Feel free to start it, but it's running already the config should be automatically reloaded!

Configuration (with the authentication token) as saved in "/etc/gitlab-runner/config.toml"
>

The Runner is now ready to be used. Nevertheless, I can’t use it right now, as it’s a plain Java image, I need some dependencies, mainly Maven. I will install it in my Runner first.

>
> cd /opt
> wget https://dlcdn.apache.org/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz
> tar -xvf apache-maven-3.8.6-bin.tar.gz
>

My Runner contains now Java and Maven. I will add now some tags to my Runner to describe it’s usage.

The tags will define on which Runner my job can be run. So I will add some to my pipeline configuration.

test-all:
  stage: test
  tags: [maven]
  except:
    - master
    - main
  script:
    - mvn clean install

I’ve added the maven tag to the jobs which need to use Maven (test-all, test-unit). The deploy job doesn’t need anything, so I add no tag. As I have only one Runner this seems to be useless. But if I have several Runners, with different configurations, this becomes useful. I can have three runners a plain Linux one, a Java one, and a Node.js one. Then, in my pipeline configuration, I indicate which Runner I want for each job.

More Gitlab-CI keywords

The before_script section is a section that will be run before each job (if added at the root level) or before a single job (if added inside a job definition).

before_script:
  - PATH=/opt/apache-maven-3.8.6/bin:/usr/local/openjdk-11/bin:$PATH
  - JAVA_HOME=/usr/local/openjdk-11

I’ve used a single Runner on my machine. Nevertheless, in companies, it use to be shared Runners and auto scaled Runners. The Runners will be available in some Kubernetes like architecture. Having the runners in a Docker environment, I can’t install the dependencies as I did with Maven. Instead, I have the image keyword. The image keyword allows me to indicate at the job level or at the pipeline level which Docker image to use. Then, I can make some specific installations in the before_script section if needed. As before, the image keyword can be added at the root level, which means that this image will be used for all the jobs, or at a job definition level, which means that this image will be used only for the current job.

image: openjdk:11.0.16-jdk

I also have the service keyword, which works similarly to the image keyword. With the service keyword, I can create a Docker image which will be available to a specific job. I can have my unit tests running in Java, and have a Postgres service next to it to run the integration tests.

  services:
    - name: postgres:13
      alias: postgres

Conclusion

• I’ve created a Gitlab CI YAML file in the project which will contain the pipeline configuration.
• I’ve added a job to run all the tests excluding the master branch.
• I’ve added another job to only run the unit tests but only for the master branch.
• I added one last job to fake deploy my application. This last job has a dependency with the unit test job.
• I’ve saved the JAR file as an artifact to have it available from the Gitlab interface.
• I’ve added the tags to select the Runner I want.
• And I’ve created my specific runner on my machine.

References

• https://docs.gitlab.com/ee/ci/
• https://git-scm.com/
• https://github.com/features/actions

Repository