Monitoring tells you **that** something is wrong. Observability is whether you can work out **why** without shipping new code — and the difference is what you can ask, not how many dashboards you have.
Three kinds of telemetry, each answering a question the others cannot.
METRICS LOGS TRACES
─────── ──── ──────
numbers over time discrete events one request's path
"5% of requests fail" "user 8821: timeout "auth 3ms → db 1.9s
connecting to db" → render 40ms"
cheap, aggregated expensive, detailed sampled, structural
ANSWERS ANSWERS ANSWERS
is something wrong? what exactly happened? WHERE is the time going?
is it worse than to this one request which service is at fault
last week?
PULL (Prometheus) PUSH (OTel, StatsD, Datadog agent)
───────────────── ─────────────────────────────────
monitoring system scrapes app sends to a collector
/metrics on a timer │
│ │
├─ failing to scrape IS ├─ works behind NAT / firewalls
│ the health signal (up) ├─ works for short-lived jobs
├─ no client config ├─ no service discovery needed
└─ needs reachability └─ you must detect silence yourself
Neither is better in the abstract. Pull gives you a free liveness signal; push works where the monitoring system cannot reach in. Most real systems end up with both — Prometheus scraping long-lived services, and a Pushgateway or OTel collector for batch jobs and edge workloads.
Three signals in three tools is three investigations. The value appears when you can move between them:
alert fires on the error-rate metric
│ click through (exemplar links a metric sample to a trace ID)
↓
trace shows the slow span is "checkout → payments"
│ click through (trace ID is in the log labels)
↓
logs for that trace ID show the exact error and the failing dependency
The mechanism that makes this work is a shared trace ID propagated through every service and attached to logs, plus exemplars linking metric samples to traces. That is a large part of why OpenTelemetry exists: one context propagation standard across all three pillars.
What are the three pillars, and when does each one fail you?
Metrics, logs and traces — and the useful half of the answer is the failure modes, because that is what shows you have used them.
Pillar
Answers
Fails when
Metrics
Is something wrong? Is it worse than before?
You need one specific request — and high cardinality kills the system
Logs
What exactly happened, in detail
Volume and cost; aggregation is slow; useless if unstructured
Traces
Where is the latency, across services
Sampling — the interesting request may not have been kept
The order you actually use them: a metric alert says that something is wrong, a trace says where, and logs say what. Skipping to logs first is the common time-waster, because you are searching text for a problem you have not localised.
A service is slow. Walk me through using telemetry to find out why.
Go top-down, and say what each step eliminates:
① METRICS is it actually slow, and for whom?
│ p99 by endpoint, by version, by region.
│ → narrow: one endpoint? one deploy? one AZ?
↓
② COMPARE what changed? overlay a deploy marker
│ → "it started at 14:02, which is when v2.3 rolled out"
↓
③ TRACES where does the time go inside a slow request?
│ find a p99 exemplar, read the span breakdown
│ → "1.9s of 2.0s is in the payments call"
↓
④ LOGS what is that service actually saying?
│ filter by trace_id, not by free text
↓
⑤ CONFIRM fix, and watch the same p99 metric recover
What each step rules out: metrics eliminate "it is not actually slow" and narrow the blast radius; the change correlation eliminates most causes outright; traces eliminate "which service"; logs give the specific error.
Self-host Prometheus/Grafana/Loki, or buy Datadog?
It is a cost and focus question, not a technical one — say that first.
Self-hosted
Commercial
Licence cost
None
Per host / per GB / per event
Real cost
Engineer time + storage + being on call for it
The bill
Scaling
Prometheus does not scale horizontally alone — you add Thanos or Mimir
Their problem
Data control
Yours; required in some regulated environments
Leaves your network
Time to value
Weeks
Days
A defensible default: below roughly 20–30 engineers, buy. The salary cost of operating a reliable observability stack exceeds most licences, and observability is infrastructure that produces no product.
Reasons to self-host anyway: regulatory or data-residency constraints; a log volume where per-GB pricing is genuinely prohibitive; or you already have a platform team running it well.
And the hedge: instrument with OpenTelemetry regardless. It decouples your code from the backend, so the decision becomes a collector configuration change rather than touching every service — which turns a one-way door into a reversible one.