datadog

Datadog Best Practices

This guide outlines our team's definitive standards for integrating Datadog into our applications and infrastructure. Adhering to these rules ensures consistent, high-quality observability data, enabling faster debugging, better monitoring, and unified insights across our stack.

---

1. Unified Service Tagging (UST) is Non-Negotiable

Every single piece of telemetry (metrics, logs, traces, events) must include the service, env, and version tags. This is fundamental for correlating data across Datadog products and achieving true end-to-end observability. Without these, your data is effectively siloed and useless for holistic analysis.

Configuration Management

Always set these as environment variables in your deployment pipeline. This ensures consistency and reduces boilerplate.

❌ BAD: Ad-hoc tagging or missing core tags

# Inconsistent or incomplete tagging
import datadog.dogstatsd as dogstatsd
import logging

# Metric without service/env/version
dogstatsd.DogStatsd(host='localhost', port=8125).increment('my_app.requests.total')

# Log without service/env/version context
logging.info("User logged in successfully", extra={"user_id": "123"})

✅ GOOD: Centralized environment variables and automatic tag injection

# Set these in your deployment environment (e.g., Kubernetes, Docker, CI/CD)
export DD_SERVICE="my-api-service"
export DD_ENV="production"
export DD_VERSION="1.0.0"
export DD_AGENT_HOST="datadog-agent.monitoring.svc.cluster.local" # Or your specific agent host

# Python example using ddtrace and standard logging
import logging
from ddtrace import tracer, config
from ddtrace.contrib.logging.logging import DatadogLogHandler
import datadog.dogstatsd as dogstatsd

# Configure ddtrace to pick up env vars automatically.
# Ensure ddtrace is initialized early in your application lifecycle.
# For auto-instrumentation, run your app with `ddtrace-run python your_app.py`.
# If not using ddtrace-run, explicitly configure:
tracer.configure(
    hostname=config.agent.hostname,
    port=config.agent.port,
    service=config.service, # Picks up DD_SERVICE env var
    env=config.env,         # Picks up DD_ENV env var
    version=config.version  # Picks up DD_VERSION env var
)

# Configure logging to send to Datadog agent and inject trace IDs
# DatadogLogHandler automatically adds service, env, version, trace_id, span_id
handler = DatadogLogHandler(host=config.agent.hostname, port=10518) # Default log intake port
logging.basicConfig(level=logging.INFO, handlers=[handler])
logger = logging.getLogger(__name__)

# Initialize DogStatsd client once, using agent host from ddtrace config
statsd = dogstatsd.DogStatsd(host=config.agent.hostname, port=config.agent.port)

@tracer.wrap()
def process_request(request_id):
    # Metrics automatically inherit service/env/version from ddtrace config when using ddtrace-run
    # Or ensure you pass them as tags if not using auto-instrumentation for metrics.
    tracer.current_span().set_tag('request.id', request_id)
    logger.info("Processing request", extra={"request_id": request_id, "user_agent": "Mozilla/5.0"})
    statsd.increment('my_api.requests.processed', tags=[f'request_id:{request_id}'])

# Example usage
process_request("req-abc-123")