Q and A with Neil Millard - 16th April 2025 - How do we make deployments SAFER?

Session summary: a live Q&A on making deployments safer — covering practical risk-reduction techniques (smaller changes, automated testing, staged rollouts, rollback plans) that came up in open discussion, aimed at teams currently deploying manually or infrequently who want a lower-risk path to more frequent releases.

Welcome, everybody — yes, this is an open meeting, anyone can come along and throw questions at me. We've got some slides to help me along with my ramblings. Today we're talking about deployment health checks — making deployments safer with automation.

A weekend project: Safe Deploy Check

I was a bit bored at the weekend, so I threw together a tool called Safe Deploy Check just for fun. It doesn't do anything too exciting yet — it looks at your PR and checks a few things. Have you submitted any files that might contain sensitive stuff, like environment variables with keys in them? A little security check. It also checks how big the PR is, simply by counting how many files are in it, and gives a warning if it's over a certain number. There are a couple of other checks in there too. It was only thrown together briefly, just to check things at a high level — there's loads more you could check, like whether there are tests, and whether those tests have decent coverage, which this tool doesn't do yet.

Why deployment health checks matter

Bad deployments are expensive, for two reasons: it costs a lot of time to fix the mess that's just been created, and it costs time to properly fix and redeploy the thing later. And when stuff goes wrong, you get hauled into a meeting to discuss what happened and why — more time lost. If you've got customers on the other end and you cause a production outage, that's worse still — damaged reputation, damaged relationships, all because we didn't have a check first.

A simple checklist can prevent a lot of these issues — does it have tests, is the coverage good enough? I purposely wrote my check code to have close to 100% test coverage. But are they decent tests? That's the other half of the story — coverage only tells you a line of code was exercised while your tests ran, not whether the test is any good. If you haven't got tests that actually try to trigger errors, that code might still get "covered," but fail in production for a completely different edge case you never thought of — because that's what production does to us.

Pre-flight and post-deploy checks

A deployment health check comes in two halves. The pre-flight side is what we just covered. Post-deploy might be as simple as a health check endpoint — the minimum version, used by Docker, Kubernetes, ECS, or a load balancer, is just "can I reach the endpoint, does it return a 200 OK." But you can go further — is the database available, can I write to it — really exercising all the components your application depends on, so when you hit that health check, you know the app is genuinely healthy. You can also validate the environment, infrastructure, app stability, and user experience — some of this is covered by UAT, app stability can be tested with load or synthetic transactions. Generally, the more testing you have in place, and the earlier you do it ("shift left"), the more stable your deployments will be.

Key pre-deploy checks

• Infrastructure readiness — I've seen many deploys fail not because a piece of infrastructure was missing, but because the permission to reach it was missing. For example: a piece of work needs a new database table for a feature. It works fine in dev. You deploy to UAT, and it dies — worked fine in test, worked fine on my machine, why did it die? Turns out, in Terraform, we forgot to grant the app permission to write to the new table. A health check that properly exercises dependencies would catch that immediately — hit the check, it tells you "I can't write to that table," you fix the Terraform code, and you're done.
• Dependency availability — in a microservices deployment, your app might rely on other services: a database over here, an auth service over there. Put those into the health check too, so you've got a little dashboard confirming each dependency is reachable, and can spot instantly if any of them aren't.
• Secrets and config — one of the things my weekend tool checks is whether certain config files are being committed. If you're checking in an .env file, that's a bad sign — hopefully it's in your .gitignore, but accidents happen, and you don't want secrets in your repository, even a private one, because it could accidentally be made public. This is the Swiss cheese model of failure — usually it takes three or four holes lining up (a leaked secret, then a repo made public) before something really bad happens, like someone breaching your AWS account.
• Version control — regular commits (even just locally) mean you can always roll back to a previous point rather than losing a day's work. Don't worry about tidiness while committing — you can always squash commits down before opening the PR.
• Maintenance mode and user comms — for big or risky changes, a scheduled maintenance window sets expectations, so when there is a bit of downtime, you can point back to the warning you gave.

These are the red flags to catch before you push the deploy button — if your infrastructure isn't ready or something's misconfigured, it's going to bite you the moment you hit that button.

Key post-deploy checks

• Smoke tests — essentially the same as the health check, sometimes as synthetic transactions: automation (Selenium, Postman, or similar) that drives the app through a set of real user-like actions and confirms everything works.
• Log monitoring — generate plenty of logs, and monitor for errors so you get alerted the moment something looks wrong.
• Service and system metrics — measure CPU, disk usage, and so on. I've seen panicked "it's all broken" moments that turned out to just be a server that ran out of disk space — an easy fix once you know, but it took 20 minutes to diagnose because nobody was watching that metric.
• Rollback readiness — if you've ever sat in a change control meeting, someone's asked "what's your rollback plan," and someone else has said "we fix forward, that's the agile way" — which doesn't cut it if something's really broken. You want an actual way to roll back: canary deployments (deploy to a small slice, test it, and restore the rest of the fleet if it fails), blue-green (deploy everything to the idle environment, switch over, switch back if it goes wrong), or feature flags (deploy the code inertly, then flip a switch to turn the feature on, so you can turn it off just as fast, or restrict it to specific users).
• Feedback and support loops — make sure UAT and support have an established way of reporting issues back to you, and write runbooks: if this happens, here are the steps to follow to make everything happy again. It's not just about getting a deployment live, it's about keeping it alive, and rolling back to the last known good state if it isn't.

What Safe Deploy Check does today

It's a quick tool, only a couple of checks in there so far — I'd love ideas for what else to add. It doesn't yet check endpoint response, open ports, disk usage, memory/swap, or certificate expiry dates — all good ideas for the future. It's written in Python, builds a small container, and runs inside GitHub Actions, giving you an output report with a rough confidence score on whether the deployment should go ahead (100% is a green light; it usually comes back around 90%, since nothing's perfect). If it finds issues, it lists them against your PR, and it's straightforward to wire into a CI/CD pipeline as a GitHub Action — docs are in the README.

A few practical rules

Just because it worked in staging doesn't guarantee production, but it's a strong indicator — that's what staging is for. One mistake I see: people patch something directly in staging until it works, then roll that straight to production — instead, roll back out of staging, fix it properly, and redeploy through the same pipeline you'd use for anything else. Always have a rollback plan; if you don't have one, blue-green deployments are an easy one to adopt. Always check monitoring straight after a deploy — a sudden spike in error rates on a new feature is your cue to switch it off or roll back. And keep runbooks documented, so three weeks after writing a feature, you don't have to rely on memory to know what an error means and what to do about it.

The goal is confidence — not just in deploying, but in recovering fast. One of the key DORA metrics is mean time to recovery after a failure. Failures are going to happen, but if you can recover quickly enough, nobody minds much. Robinhood had an outage in March 2020, right as lockdown started and everyone piled into trading from home — the platform couldn't handle it, and disruptions stopped users executing trades, in some cases costing them real money (one user reported losing out on $15,000 because they couldn't exercise options during the outage). That led to a lot of negative press. A pre-deployment checklist or infrastructure health check might well have caught the underlying capacity issue before it caused an outage.

So: start with a manual checklist, adopt a tool like Safe Deploy Check, or write your own — it's not that hard, I did mine in a weekend. Focus on repeatability and visibility: if it's repeatable, you can automate it; if it's automated, you can make it visible, on a dashboard or as a GitHub Actions check on the PR. Any private questions, find me on LinkedIn.

Someone asked how their team measures up to deployment success, and whether they even know what the DORA metrics are — I've got a video on DORA checks on my YouTube channel if you want to dig in. Someone else asked what the biggest driver was for me putting health checks in place. Honestly, these days, with a lot of experience behind me, it's just a no-brainer — I've done enough deployments and felt enough pain that "why wouldn't you have a health check" is the default question now.

One example: I once built an API that helped manage build and deployment processes (it won an award) across three test environments and a production environment. Code would develop in one test environment, deploy automatically to a third test environment, and only if that included a passing health check would it go on to production. On several occasions, that health check on the third environment caught a problem before production deployment even happened — the team later added a pre-live environment for an extra layer of confidence too. If the health check fails on the way to staging, the automated deployment simply stops there; if it passes, it goes straight to live — no unnecessary waiting, and the code also had feature flags in it, giving even finer control over what was actually live in production.