Observability: logs, metrics, traces and OpenTelemetry

HOW IT FITS TOGETHER

KEY TERMS

Cardinality

The number of distinct label combinations for a metric. Labels like method, route template and status class are fine; user ID, raw URL path, request ID or error message create one time series per value and can take down a metrics backend. Put high-cardinality data on spans and logs instead.

Structured logging

One JSON object per event with stable field names (level, msg, service, trace_id, span_id, plus domain fields) instead of interpolated strings. It makes logs queryable and joinable with traces. Redact secrets and personal data at the logger, not in the backend.

RED and USE

RED (rate, errors, duration) describes request-driven services from the caller's view and maps directly onto SLIs. USE (utilization, saturation, errors), from Brendan Gregg, is checked per resource such as CPU, memory, disk, connection pools and queues, to explain why RED went bad.

SLIs, SLOs and error budgets

An SLI is a ratio of good events to valid events (for example, requests under 300 ms with a non-5xx response). An SLO is a target over a window, such as 99.9% over 30 days; the remainder is the error budget. Alert on burn rate: Google's SRE workbook pages when 2% of a 30-day budget burns in 1 hour (14.4x), confirmed by a 5-minute window.

OpenTelemetry pieces

The API is what libraries call, and it is a no-op without an SDK. The SDK implements sampling, processors and exporters. OTLP is the wire protocol. The Collector is a vendor-neutral pipeline of receivers, processors and exporters. Semantic conventions fix attribute and metric names, such as the stable http.server.request.duration histogram in seconds.

Context propagation

The trace ID and parent span ID travel in the W3C traceparent header (00-<trace-id>-<parent-id>-<flags>). In-process, the active span lives in a context object (AsyncLocalStorage in Node, context.Context in Go). Every hop that drops it, such as a raw queue message or a custom HTTP client, breaks the trace into pieces.

Head vs tail sampling

Head sampling decides at the root span, usually from a trace-ID ratio inherited by children (parent-based), so it is cheap and consistent but cannot know whether the request will fail. Tail sampling decides in the collector after the trace completes, so it can keep every error and slow trace, at the cost of buffering whole traces in memory and routing all spans of a trace to the same instance.

IN YOUR STACK

// instrumentation.ts: load before app code
//   OTEL_SERVICE_NAME=checkout-api node --import ./instrumentation.js app.js
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";

const sdk = new NodeSDK({
  // Defaults to http://localhost:4318/v1/traces; override with OTEL_EXPORTER_OTLP_ENDPOINT.
  traceExporter: new OTLPTraceExporter(),
  metricReaders: [new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
    exportIntervalMillis: 15_000,
  })],
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Flush buffered spans and metrics before the process exits.
process.on("SIGTERM", () => {
  sdk.shutdown().finally(() => process.exit(0));
});

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("payments.capture")

def capture(order) -> CaptureResult:
    with tracer.start_as_current_span("payment.capture") as span:
        # High-cardinality attributes are fine on spans; never as metric labels.
        span.set_attribute("order.id", order.id)
        span.set_attribute("payment.provider", order.provider)
        try:
            ref = gateway.capture(order.id, order.amount_cents)
        except gateway.CardDeclined as exc:
            # Handled here, so it never escapes the with-block: record it explicitly.
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, "card declined"))
            return CaptureResult.declined(exc.code)
        span.add_event("payment.captured", {"gateway.ref": ref})
        return CaptureResult.ok(ref)

import (
	"context"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
)

var tracer = otel.Tracer("example.com/inventory")

func Reserve(ctx context.Context, sku string, qty int) error {
	ctx, span := tracer.Start(ctx, "inventory.reserve")
	defer span.End()
	span.SetAttributes(attribute.String("inventory.sku", sku), attribute.Int("inventory.qty", qty))

	// Pass the returned ctx down so DB and HTTP spans nest under this one.
	if err := store.ReserveStock(ctx, sku, qty); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "reserve failed")
		return err
	}
	return nil
}

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  memory_limiter: { check_interval: 1s, limit_mib: 1500, spike_limit_mib: 300 }
  tail_sampling:
    decision_wait: 10s        # buffer each trace this long before deciding
    num_traces: 100000        # traces held in memory
    policies:                 # a trace is kept if any policy samples it
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow, type: latency, latency: { threshold_ms: 1000 } }
      - { name: baseline, type: probabilistic, probabilistic: { sampling_percentage: 5 } }
exporters:
  otlp_grpc:
    endpoint: tempo.observability:4317
    tls: { insecure: true }   # TLS is on by default; plaintext only inside the cluster
    sending_queue: { batch: {} }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling]
      exporters: [otlp_grpc]

import pino from "pino";
import { isSpanContextValid, trace } from "@opentelemetry/api";

export const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",
  redact: ["req.headers.authorization", "req.headers.cookie", "*.password"],
  // Runs for every line: stamp the active span so logs join their trace.
  mixin() {
    const ctx = trace.getActiveSpan()?.spanContext();
    return ctx && isSpanContextValid(ctx)
      ? { trace_id: ctx.traceId, span_id: ctx.spanId, trace_flags: ctx.traceFlags }
      : {};
  },
});

export function logCapture(orderId: string, amountCents: number, durationMs: number) {
  // Fields, not string interpolation: each one is queryable.
  logger.child({ orderId }).info({ amountCents, provider: "stripe", durationMs }, "payment captured");
}

var reqDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
	Name:    "http_request_duration_seconds",
	Help:    "HTTP request latency.",
	Buckets: []float64{.01, .025, .05, .1, .25, .5, 1, 2.5, 5},
	// Also expose a native histogram (exponential buckets, ~10% growth factor).
	NativeHistogramBucketFactor: 1.1,
}, []string{"method", "route", "code"}) // route template, never the raw path

func observe(method, route string, code int, start time.Time) {
	reqDuration.WithLabelValues(method, route, strconv.Itoa(code)).
		Observe(time.Since(start).Seconds())
}

/* PromQL
# p99 per route, classic buckets: rate each bucket, then sum by le
histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

# Same query against native histograms: no _bucket suffix, no le
histogram_quantile(0.99, sum by (route) (rate(http_request_duration_seconds[5m])))

# Error-ratio SLI
sum(rate(http_request_duration_seconds_count{code=~"5.."}[5m]))
  / sum(rate(http_request_duration_seconds_count[5m])) */

WHERE IT BITES

• Putting user IDs, raw paths, tenant IDs or error strings into metric labels. Series counts explode, queries slow down and the bill follows. Use route templates and status classes, and keep the detail on spans.
• Alerting on causes (CPU at 80%, one pod restarted) instead of symptoms. Page on SLO burn rate for user-facing SLIs and use USE metrics for diagnosis, or on-call learns to ignore alerts.
• Broken propagation: a queue message without traceparent, a hand-rolled HTTP client, a Go call using context.Background(), or an API gateway that strips the header. Each gap splits one request into unrelated traces.
• Head-sampling at 1% and expecting to find the failing request. Ratio sampling drops errors at the same rate as successes; keep errors and slow traces with tail sampling, or log the trace ID so the log at least survives.
• Logging secrets and personal data, then copying them into three backends. Redact in the logger and in a collector processor; treat telemetry stores as production data with access control and retention limits.
• Calling the SDK's shutdown never, or only on SIGINT. Short-lived jobs and serverless functions exit with spans still buffered, so the traces you need most (crashes) are the ones that never arrive.

WHEN IT BREAKS IN PRODUCTION

• You can't follow one request across services
• On-call ignores pages because most alerts are noise
• Passwords, tokens, or personal data show up in logs
• The whole server freezes during one heavy request
• One request waits on a chain of services and times out
• A brief blip turns into a long outage

SOURCES

• OpenTelemetry: Sampling (opens in new tab)Checked 2026-11-06
• OpenTelemetry: Context propagation (opens in new tab)Checked 2026-11-06
• OpenTelemetry semantic conventions: HTTP metrics (opens in new tab)Checked 2026-11-06
• OpenTelemetry Collector: Configuration (opens in new tab)Checked 2026-11-06
• Prometheus: Native histograms specification (opens in new tab)Checked 2026-11-06
• Google SRE Workbook: Alerting on SLOs (opens in new tab)Checked 2026-11-06