Datadog APM: A Deep Dive on Limitations and an Open Source ...

However, as applications scale and teams mature, many engineers start asking critical questions: *Why is our Datadog bill so unpredictable and high?* *Are we missing important data because of trace sampling?* *Are we locked into a proprietary agent that limits our flexibility?* If these questions sound familiar, you're not alone. ... Datadog APM works by using a proprietary agent installed on your hosts to automatically collect traces, metrics, and logs. While this offers a seamless, integrated experience, this convenience comes with trade-offs. Let's look at the five core challenges that often drive users to seek alternatives. ### Challenge #1: The Unpredictable Datadog Cost Model The most common pain point with Datadog is its complex and often staggering cost. Your APM bill combines a **per-host platform fee** with **usage charges for ingested spans (per GB)** and **Indexed Spans (per million, by retention)**. As costs scale with volume, a medium-sized environment with around 100 hosts can often cost between **$2,000–$5,000 per month**¹. This model makes budgeting nearly impossible and often forces teams to choose between visibility and cost control. A quick search for 'Datadog billing' on Reddit reveals numerous threads from developers frustrated with its pricing: ### Challenge #2: The Sampling Dilemma Sampling is a standard technique for managing high volumes of telemetry data, and it can be effective for monitoring broad trends. However, Datadog's reliance on aggressive sampling to make its high costs manageable creates a difficult dilemma for engineering teams. The trade-off is stark: either ingest 100% of your traces for complete visibility during critical incidents and face an exorbitant bill, or sample your data to control costs and risk losing the exact trace you need to solve a problem². This becomes particularly painful during incident response. When you're hunting for the root cause of a rare bug or trying to understand the full blast radius of an error, the one trace that holds the answer may have been discarded by the sampler. You're forced to choose between cost and completeness, a compromise that can prolong outages and increase Mean Time to Resolution (MTTR). In the datadog UI below, you can see the controls where teams are asked to set sampling rates, effectively deciding which data they are willing to lose to manage their bill: ### Challenge #3: Datadog's Proprietary Agent & Vendor Lock-In When you instrument your code with the Datadog Agent, you're tying yourself to their proprietary ecosystem. This is a critical point because while Datadog can ingest OpenTelemetry data, many of its advanced APM features still require the use of its proprietary agent to function fully. This makes a full migration away from Datadog a complex task involving re-instrumenting your applications. ### Challenge #4: Technical and Data Limits in Datadog APM Beyond strategic challenges, teams can run into practical limitations around data volume and cardinality in Datadog. While the platform doesn’t enforce a strict technical cap on the number of tag combinations, high-cardinality metrics can quickly become costly and harder to query at scale. Datadog manages this through features like **Metrics without Limits™**, which lets teams drop or restrict certain tags from indexing to control performance and cost. This means data isn’t “rejected” outright, but high-cardinality tags (such as user IDs or request IDs) may not be fully indexed or queryable. For teams that rely on deep per-user or per-request granularity, this can limit the visibility they expect. Additionally, each Datadog agent consumes CPU and memory on its host or pod, creating measurable overhead in resource-constrained environments. ### Challenge #5: Limited Customizability of the Datadog APM As a closed SaaS platform, Datadog offers limited flexibility for custom needs. Users cannot modify how telemetry data is processed beyond what the platform allows. This means if you have unique instrumentation requirements or need to monitor a technology that isn't supported out-of-the-box, you must rely on Datadog's roadmap to add that support. The platform’s internals are a black box, which can be restrictive for teams with advanced or specific observability needs.