datadog

Datadog is an observability platform for monitoring applications and infrastructure. The datadog plugin sends gateway request metrics over UDP to DogStatsD, the metrics aggregation service bundled with the Datadog Agent. The Agent aggregates the metrics and forwards them to Datadog for querying, dashboards, and alerts.

Metrics​

The plugin exports the following metrics. Each name is prefixed by the namespace configured in plugin metadata. With the default apisix namespace, request.counter is exported as apisix.request.counter.

Tags​

The plugin attaches the following tags when their values are available:

Examples​

The examples deploy a pinned DogStatsD Agent, configure the gateway to reach it, and verify gateway metrics in Datadog. The Docker setup uses a dedicated network so the gateway can send UDP metrics to the Agent by container name.

Create or select a Datadog API key and identify the Datadog site for the organization. Set them as environment variables without adding the key to configuration files or source control:

export DD_API_KEY=replace-with-datadog-api-key
export DD_SITE=datadoghq.com

The Agent deployments below are intended for evaluation. Follow the Datadog Agent container guidance and the organization's secret-management requirements for production deployments.

export GATEWAY_CONTAINER=replace-with-gateway-container-name

docker network create gateway-datadog-net
docker network connect gateway-datadog-net "$GATEWAY_CONTAINER"

docker run -d \
  --name dogstatsd-agent \
  --network gateway-datadog-net \
  -e DD_API_KEY \
  -e DD_SITE \
  -e DD_HOSTNAME=apisix-docs-datadog \
  -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
  registry.datadoghq.com/dogstatsd:7.83.1

until [ "$(docker inspect -f '{{.State.Health.Status}}' dogstatsd-agent)" = "healthy" ]; do
  sleep 2
done

kubectl create secret generic datadog-api-key \
  --namespace aic \
  --from-literal=api-key="$DD_API_KEY"

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: aic
  name: dogstatsd-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dogstatsd-agent
  template:
    metadata:
      labels:
        app: dogstatsd-agent
    spec:
      containers:
        - name: dogstatsd-agent
          image: registry.datadoghq.com/dogstatsd:7.83.1
          env:
            - name: DD_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-api-key
                  key: api-key
            - name: DD_SITE
              value: datadoghq.com
            - name: DD_HOSTNAME
              value: apisix-docs-datadog
            - name: DD_DOGSTATSD_NON_LOCAL_TRAFFIC
              value: "true"
          ports:
            - name: dogstatsd
              containerPort: 8125
              protocol: UDP
          readinessProbe:
            exec:
              command:
                - /probe.sh
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: dogstatsd-agent
spec:
  selector:
    app: dogstatsd-agent
  ports:
    - name: dogstatsd
      port: 8125
      targetPort: dogstatsd
      protocol: UDP

kubectl apply -f dogstatsd.yaml
kubectl rollout status -n aic deployment/dogstatsd-agent

Configure the DogStatsD Destination​

The plugin metadata defines the DogStatsD endpoint, metric namespace, and tags shared by every route that enables the plugin. Configure the metadata for the selected environment:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/datadog" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "dogstatsd-agent",
    "port": 8125,
    "namespace": "apisix_docs",
    "constant_tags": [
      "source:apisix",
      "docs_example:datadog"
    ]
  }'

adc dump -o adc-metadata.yaml --with-id \
  --include-resource-type plugin_metadata

plugin_metadata:
  # Keep all other plugin metadata entries from the exported file.
  datadog:
    host: dogstatsd-agent
    port: 8125
    namespace: apisix_docs
    constant_tags:
      - "source:apisix"
      - "docs_example:datadog"

adc diff -f adc-metadata.yaml \
  --include-resource-type plugin_metadata

adc sync -f adc-metadata.yaml \
  --include-resource-type plugin_metadata

datadog:
  host: dogstatsd-agent.aic.svc
  port: 8125
  namespace: apisix_docs
  constant_tags:
    - "source:apisix"
    - "docs_example:datadog"

The endpoint must be reachable from the gateway. The namespace prefixes every metric, and the constant tags make the example metrics easy to identify in Datadog.

Monitor Route Metrics​

Create a route that sends request metrics to DogStatsD. The example includes the matched path, HTTP method, and a route-specific tag, and sends each metric entry to the Agent immediately for verification.

curl "http://127.0.0.1:9180/apisix/admin/routes/datadog-route" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "datadog-docs",
    "uri": "/anything/datadog",
    "plugins": {
      "datadog": {
        "batch_max_size": 1,
        "include_path": true,
        "include_method": true,
        "constant_tags": [
          "scenario:plugin-hub"
        ]
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

services:
  - name: datadog-example
    labels:
      docs-example: datadog
    routes:
      - name: datadog-route
        uris:
          - /anything/datadog
        plugins:
          datadog:
            batch_max_size: 1
            include_path: true
            include_method: true
            constant_tags:
              - "scenario:plugin-hub"
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

adc diff -f adc.yaml \
  --include-resource-type service \
  --label-selector docs-example=datadog

adc sync -f adc.yaml \
  --include-resource-type service \
  --label-selector docs-example=datadog

apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: datadog-plugin-config
spec:
  plugins:
    - name: datadog
      config:
        batch_max_size: 1
        include_path: true
        include_method: true
        constant_tags:
          - "scenario:plugin-hub"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: datadog-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything/datadog
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: datadog-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
    - type: Domain
      name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: datadog-route
spec:
  ingressClassName: apisix
  http:
    - name: datadog-route
      match:
        paths:
          - /anything/datadog
        methods:
          - GET
      upstreams:
        - name: httpbin-external-domain
      plugins:
        - name: datadog
          config:
            batch_max_size: 1
            include_path: true
            include_method: true
            constant_tags:
              - "scenario:plugin-hub"

kubectl apply -f datadog-ic.yaml

Send several requests to generate metrics:

for request_number in $(seq 1 5); do
  curl -fsS "http://127.0.0.1:9080/anything/datadog?request=${request_number}" > /dev/null
done

DogStatsD aggregates metrics on a short interval before forwarding them. In Datadog, open Metrics → Explorer, select apisix_docs.request.counter, and filter by docs_example:datadog. The metric can take a few minutes to become available in a new organization.