Not So Obvious Bugs

You are having fun eating lots of sugar and enjoying your cheesecake, and suddenly you feel a sharp pain. You go to a dentist, and they tell you that your tooth has to be removed because it’s decayed. It’s not the cavities that you can brush clean that are a threat; it’s the ones that are still hidden until it’s too late.

Well, something similar happened to me, and now I have one less wisdom tooth.

But we are talking about code today. I just miss my wisdom tooth, and I believe the message should be clear by now.

A little backstory to understand what really happened: there is this guy called Friday. His job is to procure fresh vegetables from the nearby town. Now, every day, he has to make multiple trips to the town, and I am his boss. He reports everything to me. By everything, I mean everything. For example: how many farms are supplying us with vegetables, how many types of vegetables we have, who refused to give vegetables today, how the road he took was, how many times he went to the town, and if there are any complaints from the farmers. And not just that—his duty is to check if the vegetables are fresh, what we are missing, etc., etc.

I know I am making Friday do too much, poor guy, but I promise I take good care of him. He gets the best resources to do his job, and I have provided him with a phone for any time he is in difficulty. He is supposed to call me, and I resolve the issue. See? Easy peasy. I am a good boss, I know.

One day, the road was bad. It had rained heavily, and the bridge was broken. Friday called me, but here is the twist: he had the wrong number. The call went to voicemail. Friday did his job and said everything to the voicemail. Well, it was my mistake that Friday had the wrong number. I saved it for him.

Time went by. Some days, Friday would feel unwell. Some days, the farmers would refuse to give vegetables. Some days, Friday would lose the vegetables on the way. Every time he had a problem, he would call me, the call would go to voicemail, and the problem would get lost in the depths of a random voicemail. Now it’s the end of the month. I am very happy that operations were very smooth the whole month, and we must have made staggering sales. But when I saw the numbers, they were way lower. It’s not life-threateningly bad, but bad enough to question if Friday is even doing his job properly.

I sat with the guy, explained the situation to him, and asked him why it was like that. He told me he had sent everything to voicemail and thought I knew everything. Thinking he was lying, I took his phone and checked each voicemail, and it was all there: every issue, every problem he had. Then I looked at the number, and it was wrong. Dumb me!! Right?

Now, fixing the phone number won’t magically make all the problems that Friday had disappear, but it was super important to know what problems he had. Not every problem can be solved. For example: “Hey boss, I am feeling a little depressed today,” or “I came, but not all the vegetables are ready for harvest until next week.” But in order to fix them, my goal is to know what the problems are and what is causing them before I fix them. This is the most important part of debugging.

Jumping to the solution is not the solution until you know the cause. To understand it with an example, suppose you have a dev server in the Pacific time zone and you live in IST. Now, you are practically in the future for your dev server, and you are not converting your timestamps properly. So every time the server issues you a token and you make a request with it, you get a “token invalid” error. You will think that it’s a bad token, the dev server logic is wrong, or your cookie TTL is bad.

You can fix it in several ways. Annoyed with this, you can add some leeway: “I’ll allow my token validation to tolerate this X amount of time difference.” Or you will say, “Who cares about dev?” and disable the auth middleware. Now, both of those fix the issue for you, but they do not target the problem, which was the time zone difference. In the future, it can happen to someone else. So you failed—not instantly, but eventually.

The whole point of the article was that adding robust logging to your code and setting up good alerting go a long way. If alerts don’t arrive, logs will always be your friend, telling you if the code behaved the way you expected it to. False positives are the worst, according to me, because they keep you under the illusion that everything is well until problems come knocking at your door. A combination of both helps you draw better conclusions. Until you can reproduce the error, you are in no position to resolve it, even though you think you have the solution to fix it.

I know it’s a small thing, but sometimes we are so confident in our abilities that we overlook the fact that such simple things can cause issues. It’s something I learned by making an actual mistake, so I thought I might share.