What are the most valuable insights from experienced DevOps engineers?

Quick answer: across conversations with experienced DevOps engineers, a few themes recur: automation isn't always the answer (some manual steps are cheaper to keep than to automate), communication and stakeholder skills matter as much as technical depth, tool choice should follow the team's actual constraints rather than what's trendy, and having a clear rollback plan matters more than trying to prevent every possible failure.

Hi, welcome back to the channel — today I'm talking about the six things I learned talking to 50 DevOps engineers (actually nearer 100, probably) over the course of a year. We're going to talk about why automation isn't always the answer, the unexpected skill that matters more than technical expertise, how to approach tool selection, and what to do when things go wrong.

1. Automation isn't always the answer

Should you automate everything? Mixed response — most engineers said no. So what do you automate, if not everything? DevOps engineers focus on giving business value first — understanding the pain points of the business and fixing those first. Pain points are often created by two things: manual entry going wrong (relying on a user to give correct information, which the computer then follows blindly until something breaks downstream), or a long, boring process where people just make mistakes. If we can automate those steps, we reduce the mistakes.

Automation is only worth implementing where it genuinely improves the process — there's an XKCD comic about exactly this: here's how much time you spend on a process, here's how much time is worth spending automating it. The engineers I spoke to also asked "should we" before "can we" — the first reason to automate something needs to be a good one. What sort of tasks are we talking about? Number one is the deployment process, but that's huge, so start with a small piece — maybe the software build, maybe testing, maybe creating the artifact, maybe setting up the deployment environment, or the deployment itself. Take each piece, automate it a little, and eventually you end up with a whole CI/CD pipeline once they're joined together.

2. Communication matters more than you'd think

The second thing good DevOps engineers do is communicate. Bridging the gap between development and operations is crucial. The best engineers are excellent at explaining complex problems or concepts simply. DevOps and development environments are very complicated, and the people paying you — the ones who can write the cheque — aren't necessarily as technical as you are (why else would they be hiring you?). So you need to explain, a bit like sales, the benefit of doing certain things: "yes, this takes a couple of weeks, but once it's done, your users save themselves an hour every time they do a deployment" — a cost-benefit analysis.

The concepts can get wider than that — trying to explain Kubernetes to a non-technical stakeholder, maybe just call it a magic black box, but explaining what a Docker container is might be useful if they ask. The best engineers understand the fundamentals well enough to explain step by step what each bit does. This does mean that most top DevOps engineers spend a significant chunk of their time — somewhere between a fifth and half — in meetings and documenting: meetings to shape direction, gather information, and influence, and documentation on the back end so the knowledge can be shared with whoever comes in to look after what we've put in place. Part of DevOps is planned redundancy — we come in, make stuff better, and go away again — but to go away cleanly, we need documentation so those following in our footsteps understand what we meant, how to look after the thing, and what to do when it goes wrong.

3. Tool selection

Good engineers understand principles over specific tools — knowing how a hammer, a nail, and a wall work so you can select the correct hammer, or stepping back further and realizing you actually need a screwdriver, not a hammer at all. There are lots of tools available to the DevOps community, and they can be a hindrance or a benefit depending how you use them. Great DevOps engineers can also adapt to different tech stacks and platforms — you're rarely walking into a greenfield site, there's usually stuff already there, not optimal, not working as well as it could, and you have to take that stack, those tools, and that platform, and change small bits at a time to make the whole thing run smoother. Sometimes that means ripping out a tool and replacing it, but that's usually not the first answer — understanding the tech stacks and principles well enough lets you guide the customer through incremental improvements instead.

DevOps experts also focus on patterns and practices that work across tools. If teams are siloed — often for good organizational reasons, but not necessarily the best way for them to work together — you get handoffs, and handoffs are deadly for a process: somebody has all the context on a thing, and has to somehow impart all that information to somebody else. If you're sitting right next to each other, that's easy, you can pair on it. If you're just writing a Jira ticket and expecting somebody else to read and fully understand it, that's a lot harder. Changing that pattern and practice, to remove the handoff between dev and ops, is one of the reasons DevOps exists.

4. Incident management

Having a structured approach to handling failures is critical — stuff will go wrong, I guarantee it, usually because several small things conspire together at once; any one of them alone wouldn't cause a problem, but three or four stacking up is enough chaos for something to break.

First, you've got to spot the failure — logging, monitoring, alerting — but with the right amount of alerting, so you don't go noseblind to red alerts. Once you've identified the error, is there an easy fix? Maybe we can automate fixing it, or stop it happening again. If it's a chunkier error, that's a different path — the postmortem. Postmortems are learning opportunities, not blame sessions — we don't want to know who caused it, we want to know what went wrong, when, and what we can do to fix or prevent it, and if we can't fix or prevent it, at least find a workaround that minimizes impact.

These engineers also build systems expecting failure to happen — monitoring in place to tell us when stuff goes wrong, maybe an automated attempt to fix a couple of things before a human needs to get involved. One cloud approach is making servers or containers replaceable — if one fails, it gets killed and replaced as quickly as possible, completely automated, with a record kept so we know if it happened too often or for too long and there's a root cause to investigate. Ideally, the users running on the system never notice a thing.

5. Security first

Security is built into the process from the start, not bolted on afterwards. Continuous security scanning and testing is standard practice, part of the build pipeline — we want to know everything is secure, from the hosting to the code to the libraries and dependencies, and because we've got automated testing in place, we can catch vulnerabilities before we even deploy, and fix them right away. It's a first-class citizen, a feature, not an obstacle. Keeping the right people in and the wrong people out is very important — security breaches involving usernames and passwords are horrendous PR and bad for your customers, and can genuinely lose you customers. Keep the data safe, and treat security as first class.

6. Continuous learning

This field changes rapidly, and we have to constantly adapt and learn — if you don't, you stay in the same place, or worse, get left behind. The engineers I spoke to dedicate time weekly to learning new tools and practices, staying up to date with trends — containers were once new, serverless was once new, AI may be a thing now, and whatever's coming next, we need to keep our eyes on it. As a side note, most of the people I spoke to have active side projects — partly to keep their minds from going stale, and partly as a way to test new tech, like setting up a home lab with Docker containers or a couple of graphics cards to see what AI is all about. Being able to mess about with this stuff in your own space, in your own time, keeps you fresh — it's certainly something I do a lot of.

So those are the six things I learned speaking to over 50 DevOps engineers. Got any questions? Head over to devops-answers.com, link in the description, where I'll give you the opportunity to ask me questions in a group session. I hope you found this useful — until next time, may all your deployments be smooth.