Prometheus is a metrics database with a query language. It periodically fetches numbers from your applications over HTTP, stores them with a timestamp, and lets you ask questions about how they change over time.
your app Prometheus you
──────── ────────── ───
exposes /metrics scrape stores samples PromQL "what is the
as plain text ←───── as time series ←───── error rate?"
every
15s
Logs tell you what happened in one request. Metrics tell you what is happening across all of them, cheaply enough to keep for a year.
Question
Logs
Metrics
"Why did this request fail?"
✔ the right tool
✘
"What fraction of requests fail?"
expensive to compute
✔
"Is latency worse than last Tuesday?"
practically impossible
✔
"Alert me when errors exceed 1%"
fragile
✔
A metric is a number with labels, sampled over time. That is a much smaller thing to store than a log line, which is why you can afford to keep it long enough to compare against last month.
# Requests per second, by status code
sum(rate(http_requests_total[5m])) by (status)
# Error RATIO — the shape of almost every SLO query.
# Note both halves are rates, so the units cancel.
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# p99 latency from a histogram.
# rate() first, then sum by the le label, THEN the quantile.
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# Memory per pod, top 5
topk(5, container_memory_working_set_bytes{namespace="prod"})
# A target that has stopped responding
up == 0
You do not list targets by hand. Prometheus asks the Kubernetes API what exists, and re-checks continuously.
A ServiceMonitor — the Operator way
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-app
labels:
release: prometheus # must match the Prometheus instance's selector
spec:
selector:
matchLabels:
app: my-app # which SERVICE to scrape (not the pods directly)
endpoints:
- port: metrics # the NAME of the port in the Service, not a number
interval: 15s
path: /metrics
Why does Prometheus pull instead of push? What does that cost you?
Pull means Prometheus initiates the connection on a timer. The benefits fall out of that directly:
Pull gives you
Because
A free health signal (up)
Failing to scrape is the signal. No extra check needed
No client-side configuration
The app exposes text; it does not know Prometheus exists
No app-side buffering or backpressure
The app never queues metrics or blocks on a slow backend
Trivial local debugging
curl localhost:9090/metrics shows exactly what Prometheus sees
What it costs:
Short-lived jobs may finish between scrapes and never be seen. That is what the Pushgateway exists for — and it is the only correct use of it, because a Pushgateway holds values forever and destroys the up signal.
Targets must be reachable from Prometheus. Behind NAT, in a customer's network, or on a serverless platform, that may be impossible.
Resolution is bounded by the scrape interval. A 15-second interval cannot see a 2-second spike, which is why an OOM kill often does not appear on the graph at all.
What is a cardinality explosion, and how would you find one?
Every unique combination of label values is one time series, held in memory. A label whose values are unbounded therefore multiplies series without limit until Prometheus exhausts RAM.
# Which metric names have the most series?
topk(10, count by (__name__)({__name__=~".+"}))
# Total series — the number to watch
prometheus_tsdb_head_series
# Which label is responsible, for one metric
count(count by (user_id) (http_requests_total))
The usual culprits: user IDs, request IDs, session IDs, email addresses, full URL paths containing IDs, and raw error strings.
The test to state:if you cannot enumerate every possible value of a label, it does not belong in a label. Route templates (/orders/:id) are fine; filled paths (/orders/8821) are not.
High-cardinality detail belongs in logs or traces, which are designed for exactly that and do not hold it in memory.
What should you alert on: CPU usage, or the error rate?
The error rate — and more generally, symptoms rather than causes.
Alert on what the user experiences: request failures, latency, and budget burn rate. Alert on causes only when they are both actionable and not yet visible as a symptom — a disk that will fill in four hours, a certificate expiring in a week.
Alert on
Why
Error ratio, latency, burn rate
The user feels these. They are the point
up == 0 for a whole job
Nothing is being measured; you are blind
Disk filling within N hours
Actionable before it becomes a symptom
Certificate expiry
Predictable, and preventable
Do not alert on
Why
CPU at 80%
Users cannot feel utilisation. It may be perfectly healthy