The Dev World – Sergio Lema

Yesterday I’ve solved more than 20 security vulnerabilities in less than an hour.

With nos access to the code of the project.

I was requested to solve many security vulnerabilities on a project I hadn’t worked in the last 3 years.

Hopefully, the project was only under maintenance. No evolution was added since the last time I’ve worked in.

The detected the vulnerabilities with a scan tool when they checked the Git repository of the project.

Nobody in the team had the knowledge to identify the source of the vulnerabilities neither how to solve them.

So, they called me. The last architect who worked on the project.

As I didn’t work for the last 3 years, my access to the source code was revoked. So, the only way to look to the source code and the vulnerabilities information was through a screen share of someone who do has access.

Here are the steps I followed to solve the vulnerabilities guiding the developer who shared its screen.

Find the problem

First, I tried to figure out which modules or libraries were impacted.

I read the report of the scan tool to find out that all the vulnerabilities came from the same module. This reduces the investigation time.

It was a cryptography module, to hash some information. It was an external library.

Solving the vulnerabilities

The first thought of the other people was: so, it’s time to make a pull request to this library. 🤦

No way.

I decided to first take a look if there is a newer version of the library. Where maybe the vulnerabilities are already solved.

Hopefully, it was the case.

It was a Python project with the dependencies managed by Poetry. This means that I can’t only change the number of the version in the dependency management file. Because Poetry generates a lock file with metadata of all the dependencies.

How do I generate this lock file without the access to the project? How do the guy who shares its screen generate this file with no development environment nor Poetry installed?

Upgrade the dependency

Let’s take the bull by the horns and install Poetry.

I installed Poetry hundred of times, to help newcomers or for my tutorials.

I guide him to the Poetry installation webpage. show him the Windows command to download and install Poetry. And test it. Easy.

But!

Poetry changed the way the lock file is written in recent versions. And the CI/CD pipeline wasn’t updated, so it uses the old version.

This means that if I install the latest version of Poetry in the laptop, the CI/CD pipeline will fail due to a too recent version of the lock file.

So, first I asked him to check which is the version installed in the CI/CD platform, and install the same one.

(I’ve fallen into this trap too many times to forget about the changes in the lock file of Poetry)

Once done, he only needed to upgrade the library to the latest version. Then, generate a new Poetry lock file.

Easy right?

Dependency incompatibilities

It should be easy if the project’s version of Numpy was compatible with the latest version of the cryptography library.

Upgrading Numpy to the latest version could be a solution. But Numpy was the core of the project. It would require a lot of adaptations.

I had only one hour through a screen share. And nobody wanted that.

So I’ve decided to upgrade to the first compatible version. Hopefully, no changes in the application were required.

Push the modifications

Everything seems right, it’s time to push to the Git repository.

But then, I realized that the project we modified was used as a Git submodule in many others.

This means that the problem was detected in the Git repository of the current project. But the real problem was present in all the projects which use the first one as a Git submodule.

We need to update 10 more projects.

Updating the git submodules

I’ve worked on the application for more than 2 years. Each time I made a modification in the project, I had to update the 10 other projects.

I’m lazy. I won’t do it one by one.

Hopefully, I’ve created a bash script which scans all the projects and update the Git submodule to the latest version. (I always create bash scripts)

So, we used this script to update all the projects with a single command.

Update the environments

The root project is now secure. And all the projects which use it are also secure. Work done! And in less than an hour.

Not really.

The code is more secure now. But the problem remains in production. The problem isn’t solved until it’s not solved in production.

The CI/CD pipeline triggered the deployment in the staging environment. This allowed us to test if the modifications we made impacted the application.

A quick walk through the application showed that it worked as before. No regressions.

Now it’s time to push to production. The 10 projects.

As before, I also made a script which do it automatically for all the projects.

Once the modifications were in production, we made another quick test to validate the environment.

All good now!

Conclusion

When they called me, they expected to solve the vulnerabilities alerts. But the problem goes beyond the alerts.

That’s why I insist until the fix was delivered and tested in production.

Hopefully, the vulnerabilities had no major impact, and it was already solved in the dependency we use.