Metrics, logs and traces

From the server part of your Monitoring note.

Know

Three kinds of data: metrics (numbers over time, for dashboards and alerts), logs (single events with detail, for the exact error), traces (one request across services as spans, for where the time went). Linked by the trace id in every log line.
The pipeline: the app exposes /metrics and writes JSON logs to stdout; Grafana Alloy in the cluster scrapes the metrics, tails the logs, receives traces and ships them; Grafana Cloud stores them, Mimir metrics, Loki logs, Tempo traces; Grafana queries and alerts.
Pull, not push: Alloy scrapes each pod's endpoint every 15 seconds or so and finds new pods through the Kubernetes API. No Prometheus server is needed: the data is in the Prometheus format, Mimir stores it, Grafana queries it with PromQL.
Metric types: counter (only goes up, chart its rate()), gauge (goes up and down), histogram (buckets, for percentiles with histogram_quantile). Labels must have few values; a label per URL creates millions of series.
Logs: one JSON object per line with timestamp, level, message, request id, trace id, route, status, duration; never tokens, cookies or personal data. In Loki, labels only for low-cardinality fields.
Tracing: spans with parent-child links; the W3C traceparent header carries the trace id between services; OpenTelemetry instruments the code; sample to keep cost down, but keep errors and slow traces.
What to watch and alert on: request and error rate, p95 and p99 by route group, render time versus API time, memory, restarts, event loop lag, the cache hit ratio; alert on symptoms (errors, latency, crash loops), not on CPU.

Fact check of your notes

Gotcha· The app exposes metrics; the collector does the work

"The pod monitors data and exposes it on /metrics": the app only reports its current numbers when asked. Collecting, adding labels, sending and storing is the job of Alloy and Grafana Cloud, which is why the app needs no monitoring credentials.

Exercise

Add prom-client to a local Nuxt app with a request counter and a duration histogram, run Grafana Alloy, Prometheus or Mimir and Grafana with Docker Compose, and chart the p95 of http_request_duration_seconds per route group.
Write your alert rules for a storefront, with thresholds and durations, and say why CPU is not one of them.
Switch the server logs to pino with a request id and trace id on every line, and find one request in Loki by its id.

Be able to say

Be able to say· What is the difference between metrics, logs and traces?

"Metrics are numbers over time, like error rate or p95 response time: cheap, so they drive dashboards and alerts, and they tell me something is wrong. Traces follow one request through every service as timed spans, so they tell me where the time goes. Logs are the individual events with detail, so they tell me exactly what happened. I go in that order: an alert on a metric, then the slow traces, then the logs of the failing step, linked by the trace id in every log line."

Be able to say· How would you monitor a Nuxt storefront in production?

"Metrics from every pod, collected by Alloy into Mimir or Prometheus, and dashboards in Grafana. I watch three groups: traffic and errors, so request rate and the 5xx share; speed as p95 and p99 per route group, split into render time and time waiting on APIs; and resources, memory against the requests, restarts, event loop lag and the cache hit ratio, which drops before pages get slow. Logs go as JSON to Loki, traces to Tempo. On top of that, real-user Core Web Vitals, because the server can be fine while the page feels slow. Alerts go on symptoms: error rate, latency and crash loops, not on CPU, which is the autoscaler's job."

Your notes

Open in Notes

Nothing yet. Notes filed here show up on this page and the section overview.