How do you track technical debt and measure DevOps delivery performance?

Quick answer: track the four DORA metrics per component, not just for the system as a whole (lead time for changes, deployment frequency, change failure rate, and recovery time), and separately track planned vs. unplanned work — reducing unplanned work (bugs, hot fixes, interruptions) is usually the fastest way to both pay down technical debt and improve your DORA numbers, since they're the same underlying problem.

You want to continuously deploy, you want to deploy faster, but there are two things in your way. There's the CAB, the Change Advisory Board — they want to slow you down and make sure everything's measured. And there's technical debt, which makes everything slower, like walking through treacle when you're developing code. So today I'm going to talk about how you can go faster, how we can work with the CAB, and how we can reduce technical debt. Let's go.

If you've got more questions like these, come over to tech-answers.club and ask me there — our community is very welcoming, very friendly, and we can help each other out.

What is the CAB?

The Change Advisory Board exists at a certain level of an organization to reduce the amount of disruption that changes give the business. At best, if disruption occurs, customers can't do what they need to do — they can't run their reports or whatever else. At worst, you're working for an e-tailer and something happens to production, meaning customers can't order stuff — you're turning money away, and the business really dislikes that. So the CAB is there so you can advise the rest of the business on what you're changing, what the benefit will be, how it could go wrong, and how you'll mitigate it if it does.

But when we're doing continuous deployment, that goes against the grain of that a little bit, so we need to demonstrate to the board that our changes are standard, routine, and automated, and that if something does go wrong we've got an automated fix straight away. Sounds easy, but how do we get there?

Planned vs unplanned work

Changing our code isn't easy — we've got several constraints pulling on us. We've got planned work, which is new features we want to put into the code, and unplanned work, which is stuff like bugs and hot fixes. We can analyze what's going on by looking at our stats and therefore reduce the unplanned work, because we don't really want to keep going back and fixing stuff that should already be fixed.

The DORA metrics

How can we analyze what's going on? From a DevOps perspective we've got DORA to help us — the DevOps Research and Assessment group, set up by Google about 10-12 years ago. They give us a bunch of statistics we can measure to work out where we are and how fast and stable we're going. They break down largely into two categories: throughput, how quickly we can get stuff done, and reliability.

Throughput measures the velocity of changes being made, and we assess it with two measurements. Change lead time is the amount of time it takes for code to be committed until it gets to production — shorter is better. Deployment frequency measures how often we deploy — once a day, once an hour, once a minute? The quicker the better, and the more routine we can make it, the more we can just keep running. After all, if a deployment only takes 5 minutes, fixing something with a hot fix and deploying it is a relatively low-stress task — if it all goes wrong, you can roll it back within 5 minutes too.

For stability, we've got two main measures. The change fail percentage is the number of deployments that fail versus the number that work, expressed as a percentage — if we do 100 deployments and one fails, our fail percentage is 1%, again lower is better. Failed deployment recovery time is very important, especially for the CAB — that's how long it takes to recover to a known good state following a failed deployment, and this hopefully is measured in seconds, not minutes (if you're in minutes, you're in trouble).

Tracking metrics per component

This is all good in theory, but how do we implement it? For each application you're deploying, you should have those four stats — not just four stats for the whole system, but four for every single component. If you've got a three-tier application — a web front end, a database, and a middle layer connecting the two — then you want four stats for each of those three components: how stable is a database change, how stable is an API or middleware change, how stable is a front-end change. You can track each one separately, since you might have different teams working on each, so being able to track each separately helps everyone know where they are.

Moving the needle

So we've got the stats — how can we improve them? This is where unplanned work comes in. As a quick refresher: planned is stuff you want to do, unplanned is stuff you didn't plan on, like hot fixes, bug fixes, and interruptions from the business. We can categorize the work, which gives us another stat showing how much work we wanted to do versus how much we didn't, and use that to get the right staff with the appropriate skills to go faster.

The important thing is that with these stats we can make things visible — put them on a dashboard, because what gets tracked gets improved. If the team can see them, they can share their own ideas about how to make things better. If we're tracking these stats, the result should be less unplanned work, so we can focus on the fun stuff — the planned work we actually want to do.

Reducing technical debt

As part of this we also want to reduce technical debt. Technical debt is where code has been implemented and doesn't really work as well as it could, because it wasn't planned that way — most technical debt isn't actually planned, it's the result of somebody doing the best work they could within the time constraints they had. But as the system grows, like a bowl of spaghetti, that tiny little bit is no longer fit for purpose and needs to be chopped out, recoded, or rewritten to make everything else better.

A good example of technical debt is a piece of code that needs to be touched every single time something changes in the codebase — maybe you've got a list somewhere that's manually typed in, and every time you add something you have to go update that list. That's an extra 5 seconds you're not getting back, but once you know the list is really long, you can probably write some code to update that list automatically when you add new functions — that's what I mean by rewriting it. If we're reducing technical debt and reducing unplanned work, we should see improvements in our DORA stats too: better stability, fewer bugs, fewer failed deployments, and a reduction in deployment time.

Common pitfalls to avoid

• Setting metrics as the goal. If you make broad statements like "every application must deploy multiple times today," chances are teams will try to game those metrics, which is probably not the result you want. Track individual metrics per application, and remember that trying to measure a complex system with just one number doesn't work — people will work out how to reduce that number without making the real change you're actually going for. Identify multiple metrics that matter, including ones with a healthy amount of tension between them.
• Letting industry standards block your progress. Some people will say "well, you need to have a CAB." That's true, but don't use that as an excuse not to bother — use it as a strength you can work with. The CAB is there to help keep business stability, and you want that stability so you can get paid, so work with it rather than treating it as a shield.
• Making disparate comparisons. You might have lots of stats, but remember those stats are quite discrete in what they're measuring — a piece of middleware software behaves differently to front-end software, so you can't really directly compare the two, especially if you're using different programming languages for each layer.
• Siloed ownership. Share all four metrics across the deployment, operations, and release teams to encourage collaboration and shared ownership. The metrics don't belong to any particular team — the teams should work together to improve the statistics. Deployment stats are the responsibility of everyone from developers through to testers, and you should avoid team competition — these stats exist so the business gets better and deployments improve, not so one team can say "our unplanned work is lower than yours." The point is to make the whole business faster, not to one-up another team.
• Measuring instead of improving. If it takes a lot of effort to come up with a measurement, spend that effort improving the thing instead, even if you don't know the exact number. For a small organization with limited time and budget, you want the improvement, not necessarily the precise measurement. If you know intuitively that doing something will make a process faster, don't worry about knowing the exact number — just make the improvement. If you know something's going to reduce deployment frequency from once every two weeks to once a week, just do it.

And that's all I have for you today. Thanks for asking the questions on the forums at tech-answers.club, and I'll see you soon in the next video. Till then, may all your deployments be smooth.