Instrument Distributed Tracing

Wire OpenTelemetry. Track requests cross microservices. Find slow spots.

When Use

• Debug latency cross many services
• Follow request flow, see service deps
• Spot slow DB queries, slow API calls inside transaction
• Tie traces to logs and metrics for root cause
• Measure end-to-end latency: user req to response
• Migrate old tracing (Zipkin, Jaeger) to OpenTelemetry
• Prove SLO compliance via latency percentiles

Inputs

• Required: List of services to instrument (languages, frameworks)
• Required: Backend choice (Jaeger, Tempo, Zipkin, vendor SaaS)
• Optional: Existing instrumentation libs (OpenTracing, Zipkin)
• Optional: Sampling strategy (percent, rate limit)
• Optional: Custom span attrs for business metadata

Steps

See Extended Examples for complete configuration files and templates.

Step 1: Stand Up Backend

Deploy Jaeger or Grafana Tempo. Receive, store traces.

Option A: Jaeger all-in-one (dev/test):

# docker-compose.yml
version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:1.51
    ports:
      - "5775:5775/udp"   # Zipkin compact thrift
      - "6831:6831/udp"   # Jaeger compact thrift
      - "6832:6832/udp"   # Jaeger binary thrift
      - "5778:5778"       # Serve configs
      - "16686:16686"     # Jaeger UI
      - "14268:14268"     # Jaeger HTTP thrift
      - "14250:14250"     # Jaeger GRPC
      - "9411:9411"       # Zipkin compatible endpoint
    environment:
      - COLLECTOR_ZIPKIN_HOST_PORT=:9411
      - COLLECTOR_OTLP_ENABLED=true
    restart: unless-stopped

Option B: Grafana Tempo (prod, scales):

# docker-compose.yml
version: '3.8'
services:
  tempo:
    image: grafana/tempo:2.3.0
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/tmp/tempo
    ports:
      - "3200:3200"   # Tempo HTTP
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "9411:9411"   # Zipkin
    restart: unless-stopped

volumes:
  tempo-data:

Tempo config (tempo.yaml):

server:
  http_listen_port: 3200

distributor:
  receivers:
    jaeger:
# ... (see EXAMPLES.md for complete configuration)

For prod with S3 storage:

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: s3.amazonaws.com
      region: us-east-1
    wal:
      path: /tmp/tempo/wal
    pool:
      max_workers: 100
      queue_depth: 10000

Got: Backend live. Ready for traces over OTLP. Jaeger UI or Grafana shows "no traces" first.

If fail:

• Ports in use? netstat -tulpn | grep -E '(4317|16686|3200)'
• Container logs: docker logs jaeger or docker logs tempo
• Test OTLP endpoint: curl http://localhost:4318/v1/traces -v
• For Tempo: check config syntax with tempo -config.file=/etc/tempo.yaml -verify-config

Step 2: Instrument Apps (Auto)

Use OpenTelemetry auto-instrumentation. Common frameworks. Minimal code change.

Python with Flask:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

# app.py
from flask import Flask
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# ... (see EXAMPLES.md for complete configuration)

Go with Gin framework:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/otel/sdk/trace
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

package main

import (
    "context"
    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/otel"
# ... (see EXAMPLES.md for complete configuration)

Node.js with Express:

npm install @opentelemetry/api \
            @opentelemetry/sdk-node \
            @opentelemetry/auto-instrumentations-node \
            @opentelemetry/exporter-trace-otlp-grpc

// tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
# ... (see EXAMPLES.md for complete configuration)

Got: Traces from instrumented services show in Jaeger UI or Grafana. HTTP requests auto-create spans.

If fail:

• Exporter endpoint reachable from app?
• Env vars set: OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317
• Turn on debug logs: OTEL_LOG_LEVEL=debug (Python), OTEL_LOG_LEVEL=DEBUG (Node.js)
• Test with simple span — verify export pipe
• Check for version conflicts across OTel packages

Step 3: Add Manual Instrumentation

Custom spans for business logic, DB queries, external calls.

Python manual spans:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def process_order(order_id):
    # Create a span for the entire operation
# ... (see EXAMPLES.md for complete configuration)

Go manual spans:

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
# ... (see EXAMPLES.md for complete configuration)

Span attrs best practice:

• Use semantic conventions: http.method, http.status_code, db.system, db.statement
• Business context: user.id, order.id, product.category
• Resource IDs: instance.id, region, availability_zone
• Record errors: span.RecordError(err) + span.SetStatus(codes.Error, message)
• Events for milestones: span.AddEvent("cache_miss")

Got: Custom spans in trace view. Parent-child right. Attrs visible in span details. Errors highlighted.

If fail:

• Context propagation: parent span context passed to child?
• Span names descriptive, follow naming conventions?
• Spans ended? (defer span.End() in Go, with blocks in Python)
• Attr types: strings, ints, bools, floats only
• Semantic conventions: use standard attr names where applicable

Step 4: Wire Context Propagation

Trace context must flow cross service boundaries, async ops.

HTTP headers propagation (W3C Trace Context):

# Client side (Python with requests)
import requests
from opentelemetry import trace
from opentelemetry.propagate import inject

tracer = trace.get_tracer(__name__)
# ... (see EXAMPLES.md for complete configuration)

// Server side (Go with Gin)
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
)

# ... (see EXAMPLES.md for complete configuration)

Message queue propagation (Kafka):

# Producer
from opentelemetry.propagate import inject
from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=['kafka:9092'])

# ... (see EXAMPLES.md for complete configuration)

# Consumer
from opentelemetry.propagate import extract

def process_message(msg):
    # Extract trace context from Kafka headers
    headers = {k: v.decode('utf-8') for k, v in msg.headers}
    ctx = extract(headers)

    # Continue the trace
    with tracer.start_as_current_span("process_order_event", context=ctx):
        order_id = json.loads(msg.value)['order_id']
        handle_order(order_id)

Async ops (Python asyncio):

import asyncio
from opentelemetry import trace, context

async def async_operation():
    # Capture current context
    token = context.attach(context.get_current())
    try:
        with tracer.start_as_current_span("async_database_query"):
            await asyncio.sleep(0.1)  # Simulated async work
            return "result"
    finally:
        context.detach(token)

Got: Traces span many services. Trace IDs consistent cross boundaries. Parent-child preserved.

If fail:

• W3C Trace Context propagator configured? otel.propagation.set_global_textmap(TraceContextTextMapPropagator())
• Headers passed in HTTP requests?
• Kafka: headers supported by broker version (v0.11+)?
• Debug: log traceparent header value
• Use trace viz to spot broken links

Step 5: Set Sampling Strategy

Sample to cut volume and cost. Keep visibility.

Sampling strategies:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import (
    ParentBased,
    TraceIdRatioBased,
    StaticSampler,
    Decision
# ... (see EXAMPLES.md for complete configuration)

Tail-based sampling with Tempo:

In tempo.yaml:

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]
      storage:
        path: /tmp/tempo/generator/wal
        remote_write:
          - url: http://prometheus:9090/api/v1/write
            send_exemplars: true

    # Tail sampling (requires tempo-query)
    ingestion_rate_limit_bytes: 5000000
    ingestion_burst_size_bytes: 10000000

Use Grafana Tempo's TraceQL for dynamic sampling:

# Sample traces with errors
{ status = error }

# Sample slow traces (>1s)
{ duration > 1s }

# Sample specific services
{ resource.service.name = "checkout-service" }

Got: Trace volume cut to target percent. Error traces always sampled. Sampling decision in trace metadata.

If fail:

• Sampler applied before tracer provider init?
• Sampling decision attr in exported spans?
• Tail sampling: enough buffering? (ingestion_burst_size_bytes)
• Watch dropped traces: otel_traces_dropped_total metric
• Test with synthetic high-volume traffic to validate rate

Step 6: Tie Traces to Metrics and Logs

Link traces, metrics, logs. Unified observability.

Add trace IDs to logs (Python):

import logging
from opentelemetry import trace

# Custom log formatter with trace context
class TraceFormatter(logging.Formatter):
    def format(self, record):
# ... (see EXAMPLES.md for complete configuration)

Generate metrics from traces (Tempo):

# tempo.yaml
metrics_generator:
  registry:
    external_labels:
      cluster: production
  storage:
# ... (see EXAMPLES.md for complete configuration)

Makes Prometheus metrics:

• traces_service_graph_request_total - request count between services
• traces_span_metrics_duration_seconds - span duration histogram
• traces_spanmetrics_calls_total - span call counts

Query traces from metrics (Grafana):

Add exemplar support to Prometheus datasource in Grafana:

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    jsonData:
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceName: Tempo

In Grafana dashboard, turn on exemplars:

{
  "fieldConfig": {
    "defaults": {
      "custom": {
        "showExemplars": true
      }
    }
  }
}

Got: Click metric exemplar opens trace. Logs show trace IDs. Traces link to logs. Unified debugging cross signals.

If fail:

• Exemplar support on in Prometheus (v2.26+)?
• Trace ID format matches (32-char hex)?
• Metrics generator on in Tempo config?
• Remote write endpoint reachable from Tempo?
• Test exemplar queries: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) and on() exemplar

Checks

• Backend receives spans from all instrumented services
• Traces show right parent-child cross services
• Span attrs include semantic conventions + business context
• Context propagates cross HTTP and message queues
• Sampling cuts volume to target percent
• Error traces always sampled (if error-aware sampling)
• Trace IDs in app logs, right format
• Grafana shows traces linked from metrics via exemplars
• Log panels have data links to trace viewer
• Trace retention matches storage policy

Pitfalls

• Context not propagated: Forgot to pass context downstream → broken traces. Pass context explicit.
• Spans never ended: Missing defer span.End() (Go) or with blocks (Python) → open spans, memory leaks.
• Over-instrumentation: Span for every function bloats traces. Focus on service boundaries, DB calls, external APIs.
• Missing error recording: Skip span.RecordError() → lose debug info. Always record errors in spans.
• High cardinality attrs: Unbounded values (user IDs, request bodies) as span attrs → storage pain. Sample or aggregate.
• Wrong span kind: CLIENT vs SERVER vs INTERNAL mixed up → wrong service graph. Follow semantic conventions.
• Sampling before context: Sampling must respect parent context. Use ParentBased sampler.

See Also

• correlate-observability-signals - Unified debugging across metrics, logs, traces by trace ID
• setup-prometheus-monitoring - Metrics from traces via Tempo generator
• configure-log-aggregation - Trace IDs in logs for correlation
• build-grafana-dashboards - Viz trace-derived metrics and exemplars