The Dev World – Sergio Lema

In the following article, I show how to build an effective CI/CD pipeline in GitlabCI.

I show how to separate some steps depending on the branches. Which steps are mandatory and best practices.

I use a Git system like Gitflow as I’ve explained in a previous article. The current article starts with this system.

Tests

Of course, one of the main and mandatory steps in a CI/CD pipeline is the tests. But which one? The unit tests, the integration tests, the load tests? All?

In some of my projects, I’ve separated the execution of the unit tests and the integration tests. Why?

Because the unit tests ran in less than a minute, and the integration tests took over 5 minutes. This way, we run on the local machine only the unit tests, but in Gitlab we can run both.

The unit and integration tests should run on every single branch: develop, release, hotfix, master and every single feature branch.

What about the load tests? I won’t run the load tests on production.

The load tests require a web server with the application being deployed.

I used to include the load tests only in the release or pre-production environments, as they’re stable environments.

So, let’s describe the main stages of my GitlabCI pipeline:

stages:
  test
  build
  deploy
  merge

test-unit:
  stage: test
  script:
    - # run your unit tests here

test-integration:
  stage: test
  script:
    - # run your integration tests here

test-load:
  stage: test
  only:
    - release
  script:
    - # run your load tests here

Coverage

But there is another important feature of the tests: the coverage. To keep a code base clean, I must ensure a high coverage.

How do I ensure a high coverage?

I can put a threshold where below that limit, the pipeline fails.

This can be another step of the test stage. And apply it to all the branches.

But I can also track if the feature branches are decreasing the coverage.

Let’s say my limit is at 70%. Below 70%, the pipeline fails. Over 70%, everything is green.

But the project is at 80%. This means that it will always be green.

But a new feature is coming, a big feature. With no unit tests. When checking the coverage on this branch, the result is 78%. As it’s over 70%, it’s green. But that’s not good.

Ok, I must not validate the PR, as there are no unit tests. But I can check this automatically, it’s less responsibility for the validator.

How to do it? Unfortunately, there is no easy way to do it. Some tools like Sonarqube can do it. But I’ve found no easy way to do it in bash.

Here is the code to check the general coverage of the project:

test-coverage:
  stage: test
  only:
    - master
    - hotfix
    - release
    - develop
  script:
    - # run your check against coverage

Build

In the build stage, I create the Docker image and push it to a Docker repository. Lastly, in the deploy stage, I use the previously created Docker image and use it to update my running application.

I’ve decided to separate the build from the deploy stages. Why?

Because, I want to re-use the same build stage, the same build result in many deploys.

This makes no sense when developing features in the develop branch. As the branch evolves with every commit. Each commit generates a different build, and each build must trigger a new deploy.

But let’s take a look at the pre-production environment.

Once I’ve tested my new version or my quick fix in pre-production, I want to deploy exactly the same version in production. A build may not produce the same Docker image.

What if the version of the Docker image isn’t fixed? What if the dependencies of the project are not fixed?

Those cases can generate a different version for production. And sometimes an unstable and untested version.

I’m with you, it’s rare, but I’ve seen it. That’s why I’ve decided to use the same Docker image for both pre-production and production.

build:
  only:
    - master
    - hotfix
    - release
    - develop
  stage: build
  script:
    - aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 123456789.dkr.ecr.eu-west-1.amazonaws.com
    - docker build -t $CI_PROJECT_NAME .
    - declare imageVersion=$CI_COMMIT_SHA
    - eval $(aws ecr get-login --region eu-west-1 --no-include-email)
    - docker tag $CI_PROJECT_NAME:latest 123456789.dkr.ecr.eu-west-1.amazonaws.com/$CI_PROJECT_NAME:$imageVersion
    - docker push 123456789.dkr.ecr.eu-west-1.amazonaws.com/$CI_PROJECT_NAME:$imageVersion

Deploy

Once the Docker image is built, it’s time to deploy this Docker image to my infrastructure.

Let’s say I use AWS ECS. So, an update is like changing the reference of the Docker image for a service. Here is the command:

.deploy_template: &deploy_definition
  stage: deploy
  script:
    - declare imageVersion=$CI_COMMIT_SHA
    - ARN=$(aws ecs register-task-definition --family $CI_PROJECT_NAME --container-definitions "[{\"name\":\"$CI_PROJECT_NAME\",\"image\":\"123456789.dkr.ecr.eu-west-1.amazonaws.com/$CI_PROJECT_NAME:$imageVersion\",\"cpu\":10,\"command\":[\"echo\",\"hi\"],\"memory\":10,\"essential\":true}]" | jq -r '.taskDefinition.taskDefinitionArn')
    - aws ecs update-service --cluster $AWS_PLATFORM_DEST --service $CI_PROJECT_NAME --task-definition $ARN

deploy_dev:
  <<: *deploy_definition
  only:
    - develop
  variables:
    AWS_PLATFORM_DEST: "develop"

deploy_release:
  <<: *deploy_definition
  only:
    - release
  variables:
    AWS_PLATFORM_DEST: "release"

…

The previous command is simplified, but if you want more information about how to use and deploy on AWS ECS, check this article.

Merge Back

Finally, as I may have some quick-fixes deployed only in the hotfix branch, or some stability commits deployed only in the release branch, I need a merge back stage. The merge back stage automatically merges all the missing commits to the develop branch.

I explain more deeply the need for this stage in the following article.

Here is the code used:

.merge_back_template: &merge_back_template
only:
- release
script:
- git switch $DESTINATION && git pull origin $DESTINATION
- if [[ 0 = $(git diff origin/$CI_COMMIT_BRANCH | wc -l ) ]] ; then exit 0 ; fi
- git merge "origin/$CI_COMMIT_BRANCH" --no-ff --no-edit
- rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
- git push -u origin $DESTINATION

merge_back_to_develop:
<<: *merge_back_template
only:
- release
variables:
DESTINATION: "develop"

merge_back_to_release:
<<: *merge_back_template
only:
- hotfix
variables:
DESTINATION: "release"

Conclusion

I’ve created this pipeline because it is adapted to my usage of the deployment environments and Git branches.

If you have a different workflow, feel free to adapt it and tell me in the comments what are the differences (sharing information is always valuable).