How do you set up alerting that actually gets acted on and not ignored?

The biggest problem with alerting is the sea of red. If everything is urgent, then nothing is urgent.

At the beginning there should be a sea of green checks. But this is the ideal. In reality, alerts are setup in the mist of an incident when things are on fire. A more tactical approach should be chosen. Only setup alerts on what matter most, today. No more. Until those fires are put out. Then add more alerts, gradually over time, most important to informational.

If it isn't critical, do not add it. Likewise for Errors, and Warnings. Only after the gradual build up of alerts, making them green as you go, should you attempt to model the whole application or infrastructure.

A practical starting rule: alert on symptoms users would notice — error rate, latency, availability — not on every underlying cause. A disk at 80% full is worth a dashboard entry, not a 3am page, unless it is actually about to run out.