Metric Cardinality Is Not a Prometheus Problem — It's a Pipeline Design Problem

Pre-Write Brief

Post type: Type 3 — Community Signal
Target community: r/devops, r/sre, CNCF Slack #otel, HackerNews
Hook: The OTel cardinality limits spec is getting attention this week. The take most engineers will reach for is wrong — capping at the SDK hurts you before it helps you.
Core thesis: Cardinality explosions are a pipeline design failure, not a Prometheus configuration problem. The right fix is label governance at the instrumentation layer, not a circuit-breaker at the Collector.
Credibility proof: Concrete numbers from a real Mimir migration where a single http_request_duration_seconds metric with an unbounded user_id label generated 4.2M active series and caused a compactor OOM in the first 48 hours.
Success metric: 200+ HN upvotes, gets linked in CNCF Slack #otel within 48 hours, generates 3+ inbound discovery call requests.

---

OpenTelemetry published a practical guide to metric cardinality limits this week. It covers the new CardinalityLimit field in the SDK's View API and the filter processor in the Collector. Both are genuinely useful. Neither of them is where you should start.

We have migrated eleven enterprise observability stacks off Datadog and Mimir in the last two years. Cardinality problems appear in every single one. The pattern is always the same: engineers discover the problem late, reach for the nearest knob — usually a Prometheus recording rule or a Collector filter — and treat the symptom while the root cause keeps growing. Six months later they're back, this time with a compactor that won't converge and a Grafana dashboard that takes 40 seconds to load.

This post is about where cardinality actually comes from, how to find it before it finds you, and what the OTel cardinality limits feature is actually good for (which is narrower than the spec implies).

What "High Cardinality" Actually Means at Scale

Cardinality in a time-series context is the number of unique label-set combinations for a given metric name. A metric with labels {method, status_code, region} where method has 5 values, status_code has 15, and region has 4 produces at most 300 series. That's fine.

The same metric with an additional user_id label — where user_id has 200,000 distinct values — produces up to 36,000,000 series. That is not fine.

Mimir (and Thanos, and vanilla Prometheus) store active series in memory. Mimir's ingester holds the last two hours of series in a TSDB head block. At roughly 500–800 bytes per active series in memory (varies with label name length and chunk encoding), 36M series costs 18–29 GB of ingester RAM — per replica, before replication factor.

The compactor problem is worse. Mimir's compactor merges TSDB blocks across the retention window. High-cardinality metrics produce blocks with enormous index sizes. A single metric with 4M active series can produce a block index that takes 45 minutes to compact on an m6g.2xlarge. During that window, query performance degrades across the entire tenant.

We hit this exact scenario during a migration for a SaaS company in Q1 2026. Their application was emitting http_request_duration_seconds with user_id as a label — a pattern copied from an internal Datadog dashboard where it had been harmless because Datadog's backend handles cardinality differently (at significant cost). On Mimir, 4.2M active series from that one metric caused an ingester OOM within 48 hours of cutover. The fix took four hours to implement and three weeks to fully propagate through the retention window.

The Three Places Cardinality Explodes

Understanding where cardinality originates tells you where to intervene. There are three distinct failure modes.

1. Unbounded Label Values at Instrumentation Time

This is the root cause in 80% of cases we see. A developer adds a label that seems reasonable in isolation — endpoint, tenant_id, trace_id, error_message — without understanding that the value space is unbounded in production.

trace_id as a metric label is the worst offender. Every request gets a unique trace ID. If you emit a counter with trace_id as a label, you get exactly one series per request, which means your metric cardinality grows linearly with your request rate. At 1,000 req/s, you add 86.4M new series per day. Mimir will not survive this.

error_message is the second worst. Stack traces, SQL query strings, and filesystem paths all end up as label values when developers instrument error paths naively.

2. Label Explosion from Kubernetes Metadata

The OTel Collector's k8sattributes processor is powerful and dangerous in equal measure. It can attach pod labels, annotations, and node labels to every metric. If your application pods have labels like deploy-timestamp, git-sha, or helm-chart-version — all common in GitOps workflows — and you forward those to metrics without filtering, you get a new set of series for every deployment.

A team deploying 50 times per day with 10 pods per deployment and a 30-day retention window accumulates 15,000 distinct git-sha values. Multiplied across a metric set of 200 metrics, that's 3M additional series from metadata alone.

3. Aggregation Gaps in the Pipeline

This one is subtle. You have a metric that's fine at the service level — 50 series per instance. But you're running 200 instances, and your Collector is forwarding raw per-instance metrics to Mimir instead of pre-aggregating them. You now have 10,000 series where 50 would have been sufficient for every dashboard and alert you actually run.

The fix here is a recording rule or a Collector metricstransform aggregation, not a cardinality limit.

What the OTel Cardinality Limits Spec Actually Does

The new CardinalityLimit field in the OTel SDK's View API caps the number of distinct attribute combinations a metric can accumulate within a single collection interval. When the limit is hit, new attribute combinations are mapped to an overflow series with an otel.metric.overflow=true attribute.

In the Collector, the filter processor and the newer transform processor can drop or replace attribute values before export.

Here's a minimal example of using a View in the Go SDK to cap cardinality on a histogram:

// OTel Go SDK 1.x — set a cardinality limit on a specific instrument
import (
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/metric/metricdata"
)

view := metric.NewView(
    metric.Instrument{
        Name: "http.server.request.duration",
    },
    metric.Stream{
        // Drop user_id from the attribute set entirely
        AttributeFilter: attribute.NewDenyKeysFilter("user_id", "trace_id"),
        // Hard cap: after 1000 unique attribute combos, overflow
        CardinalityLimit: 1000,
    },
)

provider := metric.NewMeterProvider(
    metric.WithView(view),
    metric.WithReader(exporter),
)

And a Collector-side example using the transform processor to hash a high-cardinality label into a bounded set:

# otelcol-contrib 0.104+
processors:
  transform/cap_user_tier:
    metric_statements:
      - context: datapoint
        statements:
          # Replace raw tenant_id with a bucketed tier label
          - set(attributes["tenant_tier"], "enterprise") where attributes["tenant_id"] in ["acme", "globex", "initech"]
          - set(attributes["tenant_tier"], "growth") where attributes["tenant_id"] != nil and attributes["tenant_tier"] == nil
          - delete_key(attributes, "tenant_id")

This is useful. But notice what it does not do: it does not tell you which metrics are approaching the limit, it does not alert you when overflow is occurring at scale, and it does not help you understand whether the overflow series is hiding a real signal you need.

The Right Intervention Order

Based on what we've seen across migrations, here is the order of interventions that actually works:

Items 6 and 7 are circuit-breakers. They prevent a runaway metric from killing your TSDB, but they cause data loss by design. If you're relying on them as a primary control, you have a process problem, not a tooling problem.

Finding Cardinality Problems Before They Find You

Mimir exposes a cortex_ingester_active_series metric per tenant. This PromQL query identifies the top metric names by active series count:

# Top 20 metrics by active series in Mimir — run against your Mimir metrics tenant
topk(20,
  sum by (metric_name) (
    cortex_ingester_active_series_custom_trackers{tracker="all"}
  )
)

For more granular cardinality analysis, Mimir's /api/v1/cardinality/label_names and /api/v1/cardinality/label_values endpoints (available in Mimir 2.10+) return the top label names and values by series count. You can query these directly or use Grafana's Metrics Explore view, which now surfaces cardinality data inline.

Set an alert on this:

# Mimir ruler alert — fires when any single metric exceeds 500k active series
groups:
  - name: cardinality.rules
    rules:
      - alert: MetricCardinalityHigh
        expr: |
          sum by (metric_name) (
            cortex_ingester_active_series_custom_trackers{tracker="all"}
          ) > 500000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Metric {{ $labels.metric_name }} has {{ $value | humanize }} active series"
          runbook: "https://wiki.internal/runbooks/mimir-cardinality"

Run this query weekly as part of a platform health review. In our experience, catching a metric crossing 100K series gives you weeks of runway to fix it at the instrumentation layer. Catching it at 4M gives you hours before you're paging your on-call.

Where the OTel Cardinality Limits Feature Is Actually Useful

I want to be clear that the new spec is a genuine improvement. There are two cases where CardinalityLimit in the SDK is the right tool:

Third-party libraries you don't control. If you're using an instrumented HTTP framework or database driver that emits high-cardinality labels, you can't change the instrumentation. A View with AttributeFilter and a CardinalityLimit is the correct containment mechanism.

Defense in depth for new services. Setting a conservative cardinality limit (say, 2,000 series per metric) on new services in staging catches instrumentation mistakes before they reach production. This is a good practice, similar to setting memory limits on containers — not because you expect to hit them, but because you want an early signal when something is wrong.

What it is not: a substitute for label governance, a solution to Kubernetes metadata explosion, or a way to fix a metric that's already in production with 10M active series.

This Is Not the Right Approach When...

If you are migrating from Datadog to a self-hosted stack and your Datadog bill is high partly because of high-cardinality metrics, do not assume that adding cardinality limits to the OTel pipeline will replicate Datadog's behavior. Datadog's backend handles cardinality through a combination of per-metric pricing, automatic rollups, and backend aggregation that is opaque to you. On Mimir, every series costs the same in ingester RAM and storage I/O. The economics are different.

If you have a multi-tenant Mimir deployment with per-tenant limits, setting max_global_series_per_user and max_global_series_per_metric is necessary but not sufficient. Tenants will hit limits, get 429s from the distributor, and lose data silently if you haven't set up alerting on cortex_discarded_samples_total. The limit is a guardrail, not a solution.

Where to Go From Here

If you're running Mimir or planning to migrate to it, the cardinality analysis endpoints and the active series alert above are the right starting point. Run the cardinality queries against your current stack — whether that's Datadog, Prometheus, or Grafana Cloud — before you migrate. You want to know your series count distribution before you're on the hook for the infrastructure.

The OTel Go compile-time instrumentation reaching v1 this week (also announced in the OTel blog) changes the calculus slightly for Go shops: you'll have less control over label sets in auto-instrumented code, which makes the View-based AttributeFilter pattern more important, not less.

We've built a cardinality audit into every migration engagement we run at Etalon — it's usually the first thing that surprises clients who assumed their Datadog metrics would map cleanly to Mimir series. If you're planning a migration and want to understand what your actual series footprint looks like before you commit to infrastructure sizing, that's a conversation worth having early.

The OTel cardinality limits spec is a good addition to the ecosystem. Just don't let it become the reason you skip the harder conversation about why those labels exist in the first place.