On Technical Debt and Why We Can't Have Nice Things

If there’s one thing engineers can’t stop talking about is technical debt and how much everyone feels like they are not doing enough to battle it.
”We have too much of it!” they say. “We should have a zero tech debt policy in the company!” some argue. “So much of our system is legacy code, this is horrible tech debt” they demand, wishing to rewrite it all.
It seems like tech debt is the one and only bane of good software engineering and if we were only to get rid of it we could reach coding utopia.

While I have no quarrel with those wishing to abolish technical debt and replace it all with sparkling and shiny technical… *arrr… *not debt, I guess. I think we all really need stop for a second and discuss: What does tech debt even mean? Do we all talk about the same thing when we say “tech debt”? How is tech debt even created in the first place and why do we hate it so much?

Software Economics 101

Anatomy of Tech Debt

Did you ever stop and ask yourself, why do we even call “technical debt” the way we do?

It’s a metaphor obviously.
But If we consult with the Miriam Webster dictionary:

debt (noun) 1 : something owed : OBLIGATION 2 : a state of being under obligation to pay or repay someone or something in return for something received : a state of owing*

So when we are in debt, technical or otherwise, there is a cost we are obliged to pay to someone.
Equally important, there is the value we received in return for this obligation.

When we talk about tech debt, we usually only talk about the first part of that equation. We have some form of a holistic view of how code should be written, how a system should be architected at a certain point in time and the standards we must abide to. Tech debt is usually referred to as the gap created and untreated between that view and what exists in practice.

That can mean things that have to do with long term code maintainability like how well code is structured and reflects the world of use cases it’s trying to solve, or smaller things like unit test coverage or documentation. That can mean functional things like bugs, lack of edge case handling or defensive coding protection and proper exception handling. It can also mean ongoing reliability things like non-scalable design, investigation and patching of common errors, stability issues and outdated dependencies.
Different teams will define debt differently and assign “cost” differently to different types of debt. How you or your team interprets debt will vary depending on your understanding of the business needs your code aims to serve as well as the expected standards and norms defined by your engineering culture.

As with any debt, the form of payment can come in various ways.
The better types of debt will just wait for you patiently until you are able to pay them off. That would rarely be the case however when it comes to technical debt.
For tech debt there will usually be some form of interest to be paid periodically until the debt was fully returned.

Same as you would pay some form of interest rate on your mortgage, tech debt might incur similar “hidden” ongoing costs.
That can be in the form of incidents and time spent on support for issues caused by bugs and reliability problems. That can be in the form of longer time it takes time to make changes. That can be due to code that no longer reflects well the world it aims to model or simply badly designed or tested code that is not structured for the changes we want to make in it.
The cost of tech debt can even come in the form of the happiness of people having to maintain code they simply don’t believe in.

Unlike debts that have some form of fixed interest rate and payments plan, the interest rate on tech debt is much less predictable. With time you might be able to better understand the ongoing cost of debt left unpaid, but you will never be able to truly predict the eventual cost.
Software has a nature of wear-and-tear, but unlike hardware it’s usually not driven by usage. It’s driven by changes made in the code or the ecosystem around it.
We might be able to predict how much a piece of software will be used. But it’s much harder to predict how much a specific part of the code will need to undergo changes, how much the needs from it and ecosystem will change and how those will end up affecting it’s overall state.

Which leads us to another part of the problem. We software people usually like to deal with absolutes. After all, our days revolve around spinning zeros and ones in ideally the most predictable manner possible. So having something unpredictable that can adversely affect our craft is understandably concerning for most people.
For many software engineers out there, ridding themselves of technical debt is not only about avoiding that constant feeling of obligation, but also about removing uncertainty and having more control over that which they build and own.
But as we mentioned, there’s another side to that equation.

I’ll pay tomorrow for a working MVP today

Root of All Evil

Some of this is cultural, but most adults have learned by now that living with debt if a part of life. It’s unfortunate I know, but usually when it comes to the big stuff, or even the smaller ones, we realize that the upfront investment needed to get what we want now is something we might only be able to afford further down the road by saving or even investing funds we don’t currently have.

You need somewhere to live, but you can’t afford buying a house right now - so you get a mortgage.
You want to invest in your career so you can be successful and make more money, but you can’t afford the academic studies - so you take a student loan.
You want to open a business but don’t have the bootstrap funds - so you go to the bank and you take on a business loan.
These are all types of debt. Debt is everywhere and is a foundational piece of modern economics.

In the same way, technical debt is usually created in return for gaining a certain value, even if you are not realizing it.
This value can be things like improving short term delivery or time-to-market. We might take certain shortcuts or focus on specific things while ignoring others. All in order to be able to deliver things faster, to stay competitive or simply get to valuable feedback so we can better plan ahead.
We might accumulate technical debt in exchange for making “quick and dirty” technical bets, the kind of high risks - high reward attempts that usually end up failing, but when successful might unlock new avenues for innovation.
Sadly, more often than not, we will accumulate technical debt simply because we didn’t know any better at the time of writing the code. We essentially pay for our future wisdom with the debt we accumulate at present time.

So is technical debt a bad thing? Well that depends.
If accumulated in a controlled way, tech debt can actually be a good thing. It can put a team or a company at a future advantage, help speed up to critical success milestones, or even help save time from over investing in failing ideas and iterate faster.
The issues start when engineering organizations lose control over their debt, or even worse, choose to ignore it’s existence completely.

Some things you can only ignore for so long

Same as with taking on a mortgage to buy a new house, it’s only a good idea if you have a rough understanding of how you are going to support that debt in the long run.
Can you afford the monthly payments? Will those monthly payments end up adversely affecting the quality of life you desire? Are you willing to continue paying that mortgage for decades or do you have a plan on how to pay it off early?
Teams need to think about their tech debt in similar terms. It’s ok to take on some debt, as long as you have the plan and commitment to pay it through.

Failing to do so will always end up resulting in an ever increasing creep of time and productivity loss, as the “interest payments” for the debt keep stacking up.
Kept left unchecked even longer than that, and teams might discover the Technical Debt Death Spiral: The tipping point where a team is no longer able to even sustain the cost of maintaining the existing debt, so it generates more debt in the attempt to deal with it. The equivalent of taking gray-market loans in order to be able to afford your mortgage monthly payments.

Managing Your Budget

Management is the key term here. Technical debt doesn’t necessarily need to be eliminated, it needs to be managed. The same way you (ideally) manage your personal ongoing income and expenditure to not end up bankrupting yourself and your family.
Doing that is sometimes easier said than done, but generally speaking it requires understanding three things:

• The debt you already have and how fast are you are accumulating more of it.
• The ongoing cost of existing debt, and how much of your “budget” does it consume on the day to day
• Your target for a well balanced “budget” and roadmap to reach it

The first part is usually the easiest of the three. Just ask any engineer to map the top 10 most horrible tech debts in their code base, and you’ll get a list of 11.
However, being able to spew lists of random things we are unhappy with about our code is far from being actionable. Every item of debt should have a price tag and motivation behind it.
Asking the simple question of “Why do you think that is tech debt?” will reveal a lot about people’s mindset and pain. But it’s also important to remember that we are all biased and not always able to differentiate between eye sores that bother us because we see them all the time, yet are otherwise harmless, and things that actually make us bleed on a daily basis.

Teams that have been carrying a lot of technical debt for a while often suffer from the Death by a thousand cuts syndrome. They are simply unable to see the accumulated negative affect of specific debts, so they end up focusing on the wrong ones.
Having an inventory of debt items is the first important step, but being able to properly prioritize it is much more important to make it actionable.
That’s the first exercise most teams end up failing.

You really should.. Photo by Towfiqu barbhuiya on Unsplash

When prioritizing our technical debt backlog there are many dimensions we should consider.
The first one is the actual cost the debt incurs on a daily basis. That should not be based solely on gut feelings, but at least partially based on data collected over time.
You can collect this data by having the team note or +1 debt items every-time they stumble on it on their day to day. You can also do it anecdotally by discussing it on daily meetings or team retrospectives.

Besides ongoing cost, we should also have some bets on future cost: An assessment of our debt’s “volatility factor”. A qualitative measurement to your current guess of how painful this debt could be in the near future.
This one takes more experience to gauge, but there are things that can help. Like how much change do we expect in this area of the code - more change means more future trouble. How will the cost of debt worsen with usage - the higher the cost and expected usage so will be future pains.

Once you’ve prioritized your backlog, it’s time to make a plan - a budget.
Your budget should be based on your goals for repaying technical debt, but also be aligned with what you can actually afford.
This budget should clearly define the amount of time and resources the team is willing to allocate for managing and reducing technical debt. It could be a fixed percentage of your team’s overall time, it could be a specific amount of hours or days each week. It can even be opportunistic by coupling it to your on-call rotation: If the person currently on-call is not handling urgent issues, their duty could be working on returning tech debt.

Finally, as with any long term budget plan, we should periodically check ourselves over time to see that we are on track.
Are we meeting the goals you’ve set ourselves? Are we able to create a net negative balance of debt each month, or are we just piling on new debt faster than returning existing one?

If you are not meeting your goals, ask yourself why.
Was the goal not realistic? Were the priorities not set correctly or were you not following them? And if you found yourself caught in the death spiral, creating more new debt, instead of returning the old ones - Maybe you should reconsider bigger things on how your team operates when writing new code.

There’s a lot more than can be said about technical debt, but they key thing to remember is that it does not manifest out of thin air. It’s created over time, intentionally or more often than not, inadvertently.
Whether tech debt is good or bad is usually a matter of decision.

The decision to be mindful and in control of the debt you create, by having the right coding and design practices. Be mindful of the debt you return, by having a plan and sticking to it. And by being mindful of the debt you choose to avoid dealing with, either because you currently can’t afford it, or because it’s not painful enough to currently address.