Two failure modes dominate monitoring: silence during outages, and so much noise that alerts get muted. Both come from the same mistake - alerting on causes instead of symptoms.
Alert on what users feel
Users don't experience CPU. They experience errors and slowness. Page a human only when error rate or latency on a user-facing path crosses a threshold for several minutes. Everything else - disk, memory, queue depth - is a dashboard, not a page.
- Golden signals per service: traffic, error rate, latency, saturation
- One uptime check per critical user journey, from outside your network
- Every page must be actionable; if the response is "watch it", it's not a page
Errors deserve context, not counts
An error tracker (Sentry or similar) with releases and user context turns "500 errors are up" into "this deploy broke checkout for Safari users". Wire deploys into it so every spike has a suspect.
Logs are for questions you haven't asked yet
Structured JSON logs with a request ID that flows through every service turn debugging from archaeology into a query. Retention of two to four weeks covers almost every real investigation at a fraction of "keep everything" cost.
Close the loop
After every incident, ask: did an alert fire? Was it the first signal? Write down the one alert or dashboard that would have caught it sooner - then actually add it. Observability grows by increments, not by platform migrations.