How do monitoring, logging and alerting work together in a DevOps environment?

Quick answer: monitoring collects and displays current state (is it up, is it healthy), logging records detailed events for after-the-fact investigation, and alerting turns a monitoring signal into a notification a human acts on. They work together as a pipeline: monitoring spots that something is wrong, logging tells you why, and alerting makes sure the right person finds out in time to act on it — skipping any one of the three leaves a gap.

Welcome back to the channel, I'm Neil, and today we're going to talk about monitoring and observability — what monitoring is, collecting and analyzing it, some tools to help you do the job, and alerts, or how not to end up staring at a sea of red.

What do we mean by monitoring?

I'm talking about metrics — the numbers we need to monitor. Then there are logs, which tell us more detail about what's going on, situational awareness, like errors. And tracing, a bit of the puzzle not everyone touches on, more on that in a moment. Why does it matter? In a software delivery team, fast feedback means you can fix things faster — if it's broke, fix it; if it's not broke, don't fix it. I can't stress that second part enough.

Logs

Logs are about the detail. You need to find where they live first — some on the servers, some from the application itself (someone logged in, someone's password expired, someone tried to access something in the database without permission). What gets logged is entirely up to whoever wrote the application. If you're running web servers, you might want access logs — who's accessing your application, from what IP address, which pages they visited. And error logs — arguably the most important, since knowing what the server knew when an error occurred helps developers fix the problem.

Metrics

Metrics are mainly numbers — performance numbers (how fast is it running, is it degrading), session counts (how many customers are using the application at once, since more concurrent users means more load), and error counts. Not all errors are fatal — some are just warnings, some are serious, like disk usage: a warning at 80% and a hard error at 90%, so you can avert disaster in time. And dashboards, somewhere to display all this so you can see at a glance what's going on.

Tracing

Tracing an application is more advanced — synthetic journeys, which you may already have from user acceptance testing, let you spot how fast things are going: how long each function or step took, in milliseconds. More often than not, the bottleneck turns out to be the database — how long a SQL query took, and whether it can be optimized. At a basic level, stack traces are a form of application tracing too — when something crashes, does it come with a stack trace alongside the error message? That gives a lot of clues on how to fix the thing.

Tools

We've got tools to make this easier — a data server for storage, data collection to get the information in, and data display to show it to ourselves, our users, and management.

Prometheus is widely used and very good at handling metric data — not as strong at logs, though it can do that too. Data gets published to the Prometheus server, either pulled or pushed depending on the collector, then stored in a fast database, because metrics come in fast. Something then hangs off the end to visualize it.

The ELK stack (Elasticsearch, Logstash, Kibana) is the other major option — Elasticsearch for storing the data, Logstash for gathering it, Kibana for visualizing it. Elasticsearch is better at storing logs than metrics, but it can store both.

For data collection, Prometheus relies on collectors, plus a push gateway for pushing metrics and logs in. The ELK stack uses Beats and other integrations. For visualization, Prometheus commonly pairs with Grafana (a separate package that also works with other data sources), while the ELK stack comes with Kibana, which is great for searching logs but can also display graphs and metrics.

Alerts

So what do we do with all this data — how do we know when something's gone wrong? Alerts, based on metrics, based on a baseline — what does "normal" look like. If you know what normal looks like, you know what busy and slow look like too, both useful when tuning servers and capacity. Alerts tell us when something's abnormal — often you want to know when something's too busy, so you can trigger automatic scaling, or get a human involved if it's really busy.

One thing to bear in mind: don't have too many alerts. Too much noise and people start ignoring them, and they stop delivering any value. It also means a status dashboard just looks like a sea of red, which is disheartening — only mark things red if they genuinely need attention. Nothing worse than something being red so often that everyone ignores it, and then when it's really red, nobody notices until it's too late.

In summary

Prometheus has its collectors, data server, and visualizations. The ELK stack follows a similar pattern — Beats collecting data, a server storing metrics and logs, and visualization on the end with Kibana or Grafana. I've only scratched the surface here — there are lots of other tools you might already be using, like Sensu, Nagios, AWS CloudWatch if you're in the cloud, SolarWinds if you've got budget for licenses, or Microsoft System Center if you're running a lot of Windows servers.

That's you set up and ready to go with monitoring and alerting — any more questions, put them in the comments below, and as always, may all your deployments be smooth.