Datadog APM

Oodle's Service Catalog reads APM stats: one row per service per environment, with the hits, errors and latency of each endpoint. The Datadog Agent computes those stats from the spans your services send it and delivers them to Oodle beside the traces. This page shows how to instrument your services with the Datadog tracers so that the agent has spans to work from.

Prerequisites

note

The traces endpoint and API key for your instance are shown in the APM (via Datadog) tile on the Integrations page (Oodle UI links: ap1, us1).

A Datadog Agent with APM enabled, sending traces to Oodle

The agent has to be in place before any tracer is. Install the Datadog Agent with these values, so that it sends traces to Oodle:

datadog:
  # The agent authenticates to Oodle with this key.
  apiKey: <oodle_api_key>
  apm:
    # Opens hostPort 8126 on every node for the tracers.
    portEnabled: true
  env:
    - name: DD_APM_DD_URL
      value: '<oodle_traces_endpoint>'

A host agent takes the same settings as api_key, apm_config.enabled: true and apm_config.apm_dd_url in datadog.yaml. To keep sending to Datadog as well, use the dual-shipping setup of the Datadog integration.

note

The agent sends the stats on the same endpoint as the traces, so there is no separate stats setting. If traces arrive but the catalog stays empty, the agent is not sending to the Oodle datadog_traces URL.

The agent address and the service identity on each service

Add these variables to every service you instrument. The tracer reads all four itself, in every language.

env:
  # Tracers default to localhost:8126. On Kubernetes the agent runs on
  # the node, so point them at the node instead of the pod.
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  # One catalog row per service per environment. A span with no
  # environment lands under "none", which no filter can separate.
  - name: DD_SERVICE
    value: checkout
  - name: DD_ENV
    value: production
  - name: DD_VERSION
    value: "1.4.2"

If the Datadog Admission Controller runs in the cluster, it injects DD_AGENT_HOST into pods labelled admission.datadoghq.com/enabled: "true", and reads the service, environment and version from the tags.datadoghq.com/* labels.

Instrument the service

Every Datadog tracer reads the same variables and reports to the same agent, so the prerequisites above cover every language. Python and Go are worked through here; the Datadog tracer docs cover the rest.

Python

Install the tracer

ddtrace instruments Flask, FastAPI, Django, requests, psycopg, SQLAlchemy and most other common libraries on its own.

pip install ddtrace

Start the service with ddtrace-run

The tracer has to start before the application imports its web framework. That is the only point at which its integrations can still patch the server. Started from inside the application, they find nothing left to patch, and every request is named after its HTTP method rather than its route.

# Wrap the command that starts the service.
ddtrace-run python app.py

# The same for a WSGI or ASGI server.
ddtrace-run gunicorn -w 4 app:app
ddtrace-run uvicorn app:app --host 0.0.0.0 --port 8000

In a Dockerfile:

CMD ["ddtrace-run", "python", "-m", "app.main"]

If you cannot change the command, import the tracer as the first line of the entrypoint instead:

# First line of the entrypoint, before any other import.
import ddtrace.auto

Deploy

Roll the service out with the environment variables from the prerequisites. ddtrace-run reads them itself; there is nothing to configure in code.

Go

Add the tracer

The tracer is one module. The integrations are separate modules under contrib/, one per library; net/http is the one shown here.

go get github.com/DataDog/dd-trace-go/v2/ddtrace/tracer
go get github.com/DataDog/dd-trace-go/contrib/net/http/v2

Start the tracer at the top of main

Start it before any server or client is built, so that every integration finds a running tracer.

import (
    "log"

    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func main() {
    // Service, environment, version and the agent address come from
    // DD_SERVICE, DD_ENV, DD_VERSION and DD_AGENT_HOST.
    if err := tracer.Start(); err != nil {
        log.Fatalf("start tracer: %v", err)
    }
    defer tracer.Stop()

    // ...
}

Serve through the traced mux

The catalog lists one endpoint per route, and the route only reaches the span when the router is the traced one.

import (
    "log"
    "net/http"

    httptrace "github.com/DataDog/dd-trace-go/contrib/net/http/v2"
)

// The wrapped mux names each request after its route pattern, so the
// catalog lists "GET /orders/{id}" rather than one "GET" for everything.
mux := httptrace.NewServeMux()
mux.HandleFunc("GET /orders/{id}", getOrder)
mux.HandleFunc("POST /orders", createOrder)

// Outbound calls join the same trace, which is what draws the edges
// between services.
client := httptrace.WrapClient(http.DefaultClient)

log.Fatal(http.ListenAndServe(":8080", mux))

gorilla/mux, chi, gin, echo and gRPC each have a wrapper of the same shape under contrib.

Deploy

Roll the service out with the environment variables from the prerequisites. The tracer reads them itself, so no address or service name is hard-coded.

Verify

Allow up to 5 minutes after the first request. Each service appears as a row in the Service Catalog (Oodle UI links: ap1, us1) under the environment you set in DD_ENV, with one endpoint per route, and its traces in the Traces Explorer (Oodle UI links: ap1, us1).

The stats behind the catalog are the dd_trace_stats_* metrics. This query shows one series per instrumented service:

sum by (service, env) (rate(dd_trace_stats_hits[5m]))

Troubleshooting

Symptom	Cause
Traces arrive but the catalog stays empty	The agent is not sending to the Oodle datadog_traces URL. Stats travel on the same endpoint as the traces, so set it through DD_APM_DD_URL.
Every endpoint shows as a bare HTTP method	Python: the tracer started after the web framework loaded. Go: the router is not the traced wrapper.
The service appears under environment none	DD_ENV is not set on the service. The agent's own DD_ENV does not reach the tracers.
No traces at all	The tracer cannot reach the agent on port 8126. On Kubernetes DD_AGENT_HOST is the node IP and datadog.apm.portEnabled must be true.
The agent logs 403s	Datadog products with no Oodle endpoint, such as process collection, are refused. Traces and stats are unaffected.