The Dev World – Sergio Lema

I put down a production database because I trusted my script.

10 years ago, I had to migrate data.

In a refactoring process, we added new columns in the database. So, we had to migrate the data from the old columns to the new ones.

I made a complicated script in SQL. With a long transaction. I tested it in my local database until it runs fine. After that, I asked the permission to run it on the development database. An environment only used by the 3 developers of the team. I need to make some adjustments but it runs fine. 15 to 30 minutes.

It was Friday. This night I had a flight to take, for a long weekend. So, I asked a colleague to run the script for me, this way I can leave earlier.

The thing is, that I didn’t tell him how long the script should run So, after an hour, he also left the script running while going on a weekend.

What happened?

The size of the production database was 10 times bigger than the one in the development environment. Thus, the RAM configured wasn’t 10 times.

The script accumulated the results in the RAM until it collapsed. making the production database unavailable. An e-commerce website used this database. Which makes the website unavailable for the weekend.

The biggest problem was that 80% of the sales are at the weekend. The big boss wasn’t happy to see the alerts on Friday afternoon.

Long story short, he had to stop the database, shut down the production website, and restart it all again. This took almost 1 day to figure out what caused the problem.

What was the consequence?

Hopefully, for me, I wasn’t fired.

But I learned a big lesson.

• I learned to never let my job to another one.
• I learned to never trust a script.
• I learned to put all the monitoring necessary to investigate the behavior of a script.
• I learned to automate everything, as much as possible, leading me to a lot of DevOps knowledge and opportunities.

The biggest lessons come from the biggest errors.