What DevOps practices and secrets should every software engineer know?

Quick answer: the DevOps practices most engineers underrate are: writing tests to protect against other people breaking your code, not just your own; treating infrastructure as code so environments are reproducible and reviewable; and building small, frequent changes into the habit rather than saving everything for a big risky release.

Hi, welcome back to the channel, I'm Neil Millard, the DevOps expert, here to answer any of your IT questions — if you're in IT and have a question, come ask me, there'll be a link in the comments. Today's subject is DevOps in software engineering — what the heck does that mean? Let's crack on. I'm going to talk through several things: the role of DevOps in software engineering, understanding the historical divide between development and operations, promoting a "you build it, you run it" mindset, implementing cross-functional teams, and techniques for improving collaboration and empathy, which is kind of at the core of what DevOps is about.

A little history

Skip ahead if you already know what DevOps is. The goal of DevOps was to reduce context switching and handoffs — you've had the issue before where you ring up somebody and they say "oh, hang on, wrong department, let me hand you over," which is fine, but the two minutes you spent describing your problem, you now have to repeat to the next person — there's no handoff of information between the two parties, and this wastes a lot of time. The biggest handoff in software development is where the developers finish their code and want to get it into production, but production is guarded by ops engineers who don't want anyone touching their precious servers — a bit odd, really, because without software running on them there'd be no point to the servers.

So the initial and most obvious goal of DevOps is to get developers talking to the ops people — hence the term "DevOps." If we get these two teams working together, there's less handoff — developers know exactly how the app works, server operators know exactly how the servers work, and before, there was often little understanding from either party of the other. So let's collaborate — software engineers understand servers, server engineers understand software. Crazy idea, right? That was well over ten years ago now, and you can read more about it in a most excellent book, "The Phoenix Project" — it's a bit of a story, but if you like stories, it's awesome.

A real example

How do DevOps practices actually improve stuff? They enable information to be shared between the two parties. Let me give you an example. I was at a customer site, and the developers were developing a .NET application that needed to run on two servers for reliability — if one goes down, the other picks up the slack, or even spread the load between the two. Amazing — except they'd written the application so that every time a customer's traffic moved to the other server, it would ask them to log in again. If you're halfway through a process and the server goes down, or the current server is too busy and your traffic gets routed to the other, less busy one, great — except it asks you to log in again, and you have to start the whole process over. Not a great experience, and the developers couldn't understand why it was happening.

So, with my ops-and-DevOps hat on, I went down to their office in person and sat with them to understand the problem. It turned out the fix was pretty easy — each server had a local database for cached information, and a shared database for session data. That session data was encrypted, which is fine, except each of the two servers was using a different encryption key. That meant if you jumped over to the other server, it could see you had session data, but couldn't do anything with it because it was encrypted with the wrong key. So I said, "hey, use the same encryption key across both servers," and — I know, I'm amazing — they did, we deployed it to pre-prod, it worked, rolled out to prod, and the customer experience instantly got 100% better.

This is where talking to each other and understanding things paid off — I needed to understand how the application worked, and they needed to understand how the servers worked, and working together we found the solution.

The historical divide

Operations were traditionally very protective of their servers, networks, cables, physical boxes — it's all very complicated, and how could a developer possibly understand any of that? Likewise, the developer mindset was "well, code involves libraries and source code and things, it's very complicated, why would a server person know anything about that?" I've heard software developer friends say "oh, DevOps is so boring" — usually meaning the sysadmin side of DevOps, what a lot of people call cloud engineers, or SREs, which is ultimately systems engineers who happen to store their configuration as infrastructure as code — that's not really DevOps, but I digress.

There was a mindset divide, different priorities, different goals. The goal for sysadmins was to keep the server running with as little downtime as possible — fine, until a developer comes along with a new, not-quite-tested version of their code, loads it onto the server, the ops guys try to follow the README, it doesn't install cleanly, there's a misconfiguration, and two hours later they've done a massive rollback, all on a weekend because that's when servers are least used by customers. A very poor feeling for the ops guys, who tried to do a deploy and got completely stuffed by the developers, and the developers weren't happy either, because they had their lovely code and the "stupid" ops guys couldn't even install it. This went on for years — just throwing code over the wall. Documentation got a bit better, perhaps, but no one thought to put the two teams in the same room, and from that siloed approach came the mistrust between them.

You build it, you run it

One answer to this is "you build it, you run it" — developers and ops share responsibility for both developing and running the application. This includes application health (does it actually work on the server), performance (does it work well on a server), and whether servers need upgrading or code needs rewriting — none of those decisions can be made unilaterally, they need to be joint. By working with the operational side, developers got operational experience, understanding how networking and servers work and how that impacts their code, and operations engineers could contribute to the design process — making sure that when you scale an application across more than one server, you're not doing daft things like a different encryption key per server, but sharing the same key across the whole architecture, so everything works smoothly.

Cross-functional teams

The easiest way to implement "you build it, you run it" is a cross-functional team — one team containing several developers and several ops/DevOps people, to break down the silos. The smallest team you could get away with is a team of two — one developer, one ops person, pairing together, both learning from the other, both deploying together. A cross-functional team doesn't just cross-pollinate knowledge between the two, it gives them a shared direction to face in. We can reflect this in metrics — shared goals like application uptime, which depends on both developers and sysadmins, and other metrics like server errors or application errors, which are closely linked — if both parties are looking at these, they can work together to solve them, and through knowledge sharing and joint problem-solving you build trust and build the relationship.

In a remote-working world, is that easier or harder? Not sure — we've all got webcams now, like the one I'm talking to you on, so we can set up online meetings, though it's much better to see somebody in person, so onsite or in-person meetups are good too, particularly outside of a work environment — a beer, maybe. I run a monthly tech meetup where I live, called Western Tech Meetup, you should come along.

By facilitating regular meetups and feedback loops, we ensure we're on the same path, sharing ideas, even sharing criticism after a while — if the process is failing, we should be comfortable enough to call it out and resolve it together. If you want to go radical, you can even swap roles — how well can the developer administer a server or application in production, or how about ops writing some code for the application itself? That might sound mad, but especially if you're pairing, swapping who's at the keyboard and who's observing and chipping in can foster a great understanding of each other's challenges.

Of course, something will always go wrong, no matter how well you do it — so by having a trusted, blameless postmortem, going through the events that unfolded, you can solve problems together, building trust and getting to a working solution.

In summary

That's what I've got for you today — we covered the role of DevOps in software engineering, understanding the historical divide, promoting a "you build it, you run it" mindset, implementing cross-functional teams, and techniques for improving collaboration. If you've got any questions for me, head over to devops-answers.com, or stick them in the comments below — I love answering questions, I love helping IT professionals, and we can work together and collaborate soon. Until then, may all your deployments go smooth.