OpenTelemetry Metric Cardinality Will Bankrupt Your Mimir Cluster (and How to Fix It Before It Does)

The Cardinality Problem Is Not New, But OTel Makes It Worse

Prometheus has had cardinality problems since 2016. What OpenTelemetry changes is the blast radius. When you instrument a service with the OTel SDK and wire it to a Collector pipeline, every developer on every team can now emit metrics with arbitrary label sets. The SDK does not stop them. The Collector, by default, does not stop them. And Mimir will happily ingest every unique time series until your ingestor pods OOM and your on-call engineer gets paged at 3 AM.

The OTel project published a practical guide on cardinality limits this month, which is a good sign that the community is taking this seriously. But the guide is necessarily generic. What we will cover here is specific: how cardinality actually manifests in a Mimir deployment on AWS, what the Collector configuration looks like to contain it, and where the approach breaks down.

What Cardinality Actually Costs in a Self-Hosted Stack

Before fixing anything, it helps to understand the cost model. In Mimir, each active time series consumes memory in the ingestor. The Grafana Labs published guidance puts this at roughly 1–2 KB per active series in the ingestor's TSDB head block, depending on label count and churn rate. That number is not marketing — we have measured it in production.

Here is what we observed on a migration from Datadog for a SaaS company with 12 backend services:

The jump from 1.2M to 4.8M series happened because developers started adding user_id, session_id, and request_id as metric labels once they had direct SDK access. On Datadog, those labels existed too, but Datadog's indexing model obscured the cost until the monthly invoice arrived. On Mimir, the cost is visible in real time — and it is your infrastructure bill, not a SaaS invoice.

Finding the Offenders with PromQL

Mimir exposes its own internal metrics. The most useful query for cardinality investigation is:

topk(20,
  sum by (metric_name) (
    cortex_ingester_active_series{job="mimir-ingester"}
  )
)

This gives you the top 20 metric names by active series count across all ingestors. Run this before you do anything else. In every migration we have done, the top 5 metric names account for more than 60% of total cardinality.

For drilling into label-level cardinality on a specific metric, use:

count by (__name__, job) (
  {__name__="http_server_request_duration_seconds_bucket"}
)

If that returns a number above 50,000, you have a problem metric. The usual culprits are histograms with high-cardinality labels, specifically url_path or user_agent on HTTP duration metrics.

Mimir also ships a cardinality API endpoint that is faster than PromQL for this use case:

curl -s "http://mimir-query-frontend:8080/api/v1/cardinality/label_names" \
  -H "X-Scope-OrgID: your-tenant" | jq '.cardinality | sort_by(.label_values_count) | reverse | .[0:10]'

This returns the top label names by unique value count. If user_id appears in that list, you already know what to do.

Fixing It at the Collector Layer

The right place to cap cardinality is the OpenTelemetry Collector, not the application and not the storage backend. Fixing it in the application requires coordinating across every team. Fixing it in Mimir with max_global_series_per_user just causes ingestion errors that are hard to debug. The Collector sits in the middle and can apply consistent policy without touching application code.

The transform Processor for Label Removal

The first line of defense is removing high-cardinality labels entirely. If http.url is being emitted as a metric attribute and it contains query parameters, it needs to go before it becomes a label in Mimir.

processors:
  transform/drop_high_cardinality_attrs:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "http.url")
          - delete_key(attributes, "user_agent.original")
          - delete_key(attributes, "session.id")
          - delete_key(attributes, "enduser.id")

This is blunt. You lose the label entirely. Sometimes that is the right call — a metric labeled with a full URL path is rarely useful in aggregate and almost always a cardinality bomb.

The metricstransform Processor for Label Normalization

When you want to keep a label but reduce its cardinality, normalize it. A common case is http.route — some frameworks emit the raw matched route, others emit the full path with IDs interpolated.

processors:
  metricstransform/normalize_routes:
    transforms:
      - include: http_server_request_duration_seconds
        match_type: regexp
        action: update
        operations:
          - action: update_label
            label: http_route
            new_label: http_route_normalized
            value_actions:
              - regexp: "^/api/v[0-9]+/users/[^/]+/.*$"
                new_value: "/api/v{version}/users/{id}/..."
              - regexp: "^/api/v[0-9]+/orgs/[^/]+/.*$"
                new_value: "/api/v{version}/orgs/{id}/..."

This approach works, but it requires you to enumerate the patterns. In practice, you spend one afternoon writing regexes and then revisit it every time a new service joins the platform. It is maintenance work, not a one-time fix.

The filter Processor for Metric Allowlisting

The most aggressive control is an allowlist: only emit metrics that are explicitly approved. This sounds draconian, but it is the only approach that prevents cardinality creep over time.

processors:
  filter/metric_allowlist:
    metrics:
      include:
        match_type: regexp
        metric_names:
          - "^http_server_request_duration_seconds.*$"
          - "^http_server_active_requests$"
          - "^process_runtime_jvm_memory_used$"
          - "^process_runtime_jvm_gc_duration.*$"
          - "^rpc_server_duration.*$"
          - "^db_client_operation_duration.*$"
          - "^system_cpu_utilization$"
          - "^system_memory_usage$"
          - "^kafka_consumer_lag$"

Anything not on this list is dropped at the Collector. New metrics require a pull request to the Collector configuration. This creates friction, which is the point. Engineers think twice before adding a metric with 100,000 unique label combinations.

We run this in combination with the transform processor above. The allowlist controls which metrics reach Mimir; the transform processor controls which labels survive on the allowed metrics.

Cardinality Limits in the Collector Itself (OTel v0.105+)

As of OpenTelemetry Collector Contrib v0.105.0, the metricslimit processor is available in alpha. It enforces a hard cap on unique time series per metric name within the Collector's memory:

processors:
  metricslimit:
    max_cardinality: 10000
    overflow_action: drop
    overflow_attribute:
      key: otel.cardinality_overflow
      value: "true"

When a metric exceeds 10,000 unique label combinations, new series are dropped and the otel.cardinality_overflow attribute is set on the overflow data points. You can then alert on this attribute in Grafana to identify which services are generating cardinality explosions before they hit Mimir.

I want to be honest about the current state: this processor is alpha, the semantics around what counts as a unique series are still being refined, and the overflow_action: drop behavior means you silently lose data. Use it as a circuit breaker and a signal, not as a primary control.

Mimir-Side Limits as a Backstop

Collector-layer controls are your primary defense, but Mimir's per-tenant limits are your backstop. These belong in your Mimir runtime.yaml or per-tenant override configuration:

overrides:
  tenant-production:
    ingestion_rate: 150000
    ingestion_burst_size: 300000
    max_global_series_per_user: 2000000
    max_global_series_per_metric: 100000
    max_label_names_per_series: 30
    max_label_value_length: 1024

max_global_series_per_metric: 100000 is the one that catches the worst offenders. If a single metric name generates more than 100,000 unique series across your entire tenant, Mimir will reject the excess with a 429 and log the metric name. That log line is your alert.

Set max_label_names_per_series: 30 conservatively. OTel resource attributes get promoted to labels by default in the Prometheus remote write exporter. A service with a verbose resource attribute set can easily hit 40+ labels per series, and each additional label multiplies cardinality.

Where This Approach Breaks Down

This is not the right approach in three situations.

High-cardinality data that is genuinely useful. If your business actually needs per-user latency breakdowns, metric cardinality is not your problem to solve — you need a different data model. Use Tempo traces with exemplars and query them with TraceQL. Metrics are for aggregates. Traces are for individual requests. Trying to make metrics do per-user analysis is the wrong abstraction, and no amount of Collector configuration will make it right.

Rapidly evolving services during early development. Allowlisting works well for stable, production services. For a service that is changing its instrumentation weekly, an allowlist creates constant friction with the team building it. We typically exempt new services from the allowlist for their first 90 days in production, with a cardinality budget of 50,000 series, and enforce the allowlist only at the point of declaring the service stable.

Multi-tenant platforms where you do not control the application code. If you are running a platform where tenants bring their own services, you cannot rely on Collector-layer controls alone because each tenant may run their own Collector. In that case, Mimir's per-tenant limits are your primary control and you need per-tenant dashboards showing cardinality trends so you can intervene before limits are hit.

The Alerting You Need

Once controls are in place, you need alerting to detect cardinality growth before it becomes a crisis. Two alerts cover most cases:

Alert 1: Series count growth rate

# Fires when total active series grows more than 20% in 24 hours
(
  sum(cortex_ingester_active_series{job="mimir-ingester"})
  /
  sum(cortex_ingester_active_series{job="mimir-ingester"} offset 24h)
) > 1.2

Alert 2: Per-metric series approaching limit

# Fires when any metric is at 80% of the per-metric series limit
topk(5,
  sum by (metric_name) (cortex_ingester_active_series_per_metric{job="mimir-ingester"})
) / 100000 > 0.8

Adjust the 100000 divisor to match your max_global_series_per_metric setting.

The Operational Workflow

Cardinality management is not a one-time fix. It is an ongoing operational practice. What works in production:

1. Weekly cardinality review. A 15-minute look at the top-20 metrics by series count. Anything that grew more than 10% week-over-week gets a ticket to investigate.
2. Pre-deployment cardinality check. Add a CI step that lints OTel SDK configuration for known high-cardinality patterns (user_id, session_id, request_id as metric attributes). This is a grep, not a sophisticated analysis, and it catches 80% of the problems.
3. Collector config as code with review. Every change to the allowlist or transform processors goes through pull request review. The reviewer checks whether the new metric or label is necessary and whether it has a bounded cardinality.
4. Tenant-level cardinality dashboards. A Grafana dashboard per team showing their active series count, top metrics by cardinality, and trend over the last 30 days. Make the cost visible. Engineers who can see that their debug_request_trace_id label is responsible for 800,000 series fix it themselves.

Where to Go Next

The OTel project's cardinality limits guide is worth reading for the SDK-level controls — specifically the CardinalityLimit setting on Views in the SDK, which lets you cap cardinality at the point of emission rather than at the Collector. That is a better architectural position than Collector-layer dropping, because you are not silently discarding data; you are preventing it from being created.

The Mimir documentation on active series limits covers the full set of per-tenant controls, including the newer max_global_series_per_native_histogram_bucket limit that matters if you have migrated to native histograms (which you should, but that is a separate post).

If you are in the middle of a migration from Datadog or Splunk and you are seeing ingestor memory climb unexpectedly, the diagnosis and remediation above is the playbook we use. We have run this exact process on migrations ranging from 500K to 8M active series. The Collector configuration patterns are transferable; the specific label lists and cardinality budgets have to be calibrated to your stack.

If you want a second pair of eyes on your Mimir ingestor configuration or your Collector pipeline before cardinality becomes a production incident, Etalon does focused architecture reviews. We are not going to sell you a year-long engagement — if a two-hour review of your configuration is what you need, that is what we offer.