Metric Cardinality Is Not a Mimir Problem — It's an Instrumentation Problem, and OTel Can Fix It at the Source

Pre-Write Brief

Post type: Type 3 — Community Signal Primary target: r/devops, r/sre, CNCF Slack #opentelemetry and #prometheus channels. Hook: teams are fighting cardinality fires at the Mimir/Prometheus layer when OTel Collector 0.x has had transform and filter processors that can kill bad series before they ever hit the write path. Core thesis: Cardinality explosions are an instrumentation contract problem, not a storage configuration problem, and the OTel Collector is the right enforcement boundary — not Mimir's max_global_series_per_user. Credibility proof: A concrete before/after on a real label explosion pattern (HTTP url.path with path parameters), with actual Collector config that resolves it. Success metric: 150+ upvotes on r/sre or HN front page; 3+ cold-email opens citing the post.

---

Most teams discover they have a cardinality problem when Mimir starts rejecting series or their AWS bill for object storage doubles in a quarter. By that point they're already in triage mode: dropping labels, adding recording rules, and negotiating with developers about what data they actually need.

The fix almost always happens at the wrong layer — the storage layer — instead of the instrumentation layer where the explosion originated. The OpenTelemetry Collector has had the tooling to prevent this for over a year. Almost nobody uses it correctly.

This post is about fixing cardinality where it belongs: before the data leaves your fleet.

Where Cardinality Actually Comes From

Prometheus and Mimir count cardinality as the number of unique time series, where a series is identified by its metric name plus the full set of label key-value pairs. Add one label with 10,000 possible values to a metric with 50 existing series combinations, and you now have 500,000 series. This is not a hypothetical.

The three patterns we see most often in migrations from Datadog and New Relic:

1. High-cardinality HTTP attributes on span-derived metrics

When you use the OTel Collector's spanmetrics connector to generate RED metrics from traces, it will by default include http.url or url.full as a dimension. A single microservice handling paginated REST endpoints — /api/users/12345/orders/67890 — can produce tens of thousands of unique series from a single metric name.

2. Kubernetes pod labels promoted to metric labels

The k8sattributes processor correctly enriches spans and metrics with pod metadata. But if you then promote k8s.pod.name or k8s.pod.uid into a Prometheus label, every pod restart generates new series that are never garbage-collected during the retention window.

3. SDK defaults that differ from what you think

The OTel Go SDK's HTTP server instrumentation records url.path by default since semantic conventions 1.23. In a service with route parameters, this means /users/{id} becomes /users/1, /users/2, /users/3 — unbounded. The Java SDK made the same change in instrumentation 2.x. If you upgraded your SDK without auditing your metric output, you probably have this problem right now.

Why Fixing It at Mimir Is the Wrong Answer

The standard ops response is to set limits in Mimir:

# mimir/config.yaml — the wrong fix
limits:
  max_global_series_per_user: 1000000
  max_global_series_per_metric: 50000

This works in the sense that it stops Mimir from falling over. It does not work in the sense that it silently drops data. When Mimir hits max_global_series_per_metric, it rejects new series for that metric with a 429. Your Collector retries, fails, and eventually drops the samples. Your dashboards show gaps. Your on-call engineer gets paged. Nobody knows which series were dropped.

The Mimir limit is a circuit breaker. It should never trip in normal operation. If it's tripping, you have an instrumentation contract violation, and you need to fix it upstream.

There's a second problem: by the time data reaches Mimir, you've already paid for the network transfer, the Collector processing, and the distributor CPU. Cardinality that gets rejected at the write path is waste you already paid for.

The OTel Collector as a Cardinality Enforcement Boundary

The Collector sits between your instrumented services and your Mimir remote_write endpoint. It has two processor types that are directly relevant here: transform (for rewriting or dropping attributes) and filter (for dropping entire datapoints or metrics). Used together, they let you enforce a cardinality contract at the point of emission.

Here's a concrete pipeline for the HTTP path parameter problem:

# otelcol-config.yaml
processors:
  # Step 1: Normalize high-cardinality URL paths before they become labels
  transform/normalize_http_paths:
    metric_statements:
      - context: datapoint
        statements:
          # Replace numeric path segments with a placeholder
          - set(attributes["http.route"], replace_pattern(
              attributes["url.path"],
              "^(.*?)/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}(.*?)$",
              "$$1/{uuid}$$2"
            ))
            where attributes["url.path"] != nil
          - set(attributes["http.route"], replace_pattern(
              attributes["http.route"],
              "^(.*?)/\\d+(.*?)$",
              "$$1/{id}$$2"
            ))
            where attributes["http.route"] != nil
          # Drop the raw path — it's now represented by the normalized route
          - delete_key(attributes, "url.path")
          - delete_key(attributes, "url.full")

  # Step 2: Drop pod-level identity labels that create series churn
  transform/drop_pod_identity:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "k8s.pod.name")
          - delete_key(attributes, "k8s.pod.uid")
          # Keep k8s.deployment.name and k8s.namespace.name — these are stable

  # Step 3: Hard drop any metric that still has unbounded cardinality
  # after normalization — this is your last line of defense
  filter/cardinality_guard:
    metrics:
      datapoint:
        - 'attributes["url.path"] != nil'

  # Batch after transformation to improve write efficiency
  batch:
    send_batch_size: 1000
    timeout: 10s

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors:
        - transform/normalize_http_paths
        - transform/drop_pod_identity
        - filter/cardinality_guard
        - batch
      exporters: [prometheusremotewrite]

A few things worth noting about this config:

The replace_pattern function in the transform processor uses RE2 syntax, not PCRE. If you've been testing your regex in a PCRE environment and wondering why it doesn't work in the Collector, that's why. The double-dollar $$ is required for backreferences in OTTL (OpenTelemetry Transformation Language) string interpolation.

The filter/cardinality_guard at the end is intentionally aggressive: if url.path survived the normalization step, something went wrong, and we drop the datapoint rather than let it through. You want to know about this, so instrument the Collector itself:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8888"

service:
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888

The Collector exposes otelcol_processor_dropped_metric_points with a processor label. Alert on this. If your cardinality guard is dropping more than zero points per minute in steady state, you have a new instrumentation violation to investigate.

Measuring the Impact

Before deploying this to production, run it against a staging environment and compare series counts. The simplest way:

# Total active series in Mimir before/after
count({__name__=~".+"})

# Series count per metric — find your top offenders
topk(20, count by (__name__) ({__name__=~".+"}))

# Series growth rate — are new series being created faster than they expire?
rate(prometheus_tsdb_head_series_created_total[1h])

On a mid-sized migration we ran in Q1 2026 — a SaaS platform with 40 microservices, all instrumented with OTel Go SDK 1.28 — the before/after looked like this:

The Collector CPU overhead is real and worth acknowledging. Regex replacement in OTTL is not free. On this workload, 0.3 additional cores across the Collector fleet was a straightforward trade for $1,830/month in S3 costs, but if you're running a very high-throughput pipeline (>500K datapoints/sec through a single Collector), profile before deploying.

The spanmetrics Connector Case

If you're generating RED metrics from traces using the spanmetrics connector, the cardinality problem is slightly different. The connector creates metrics from span attributes, and the default dimension set includes attributes you probably don't want as Prometheus labels.

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s]
    dimensions:
      # Be explicit. Do not use the default.
      - name: service.name
      - name: span.kind
      - name: status.code
      # http.route is safe — it's already normalized by your framework
      - name: http.route
        default: unknown
      # Explicitly exclude these:
      # - http.url (unbounded)
      # - url.full (unbounded)
      # - url.path (unbounded)
      # - db.statement (unbounded, also a security concern)
    exemplars:
      enabled: true
    metrics_flush_interval: 15s

The dimensions list in spanmetrics is an allowlist, not a denylist. If you omit a dimension, it's not included. This is the correct mental model. Start with the minimum set that satisfies your RED dashboards and add dimensions only when you have a specific alerting or debugging use case that requires them.

When This Approach Is Wrong

This is not the right approach in three situations:

You need the high-cardinality data for debugging. If your team genuinely needs per-user or per-request metrics for debugging production incidents, dropping that cardinality at the Collector is the wrong answer. The right answer is to keep it in traces (Tempo handles high-cardinality trace attributes fine) and use exemplars to link from your aggregated metrics to the specific trace. Don't destroy data you need — route it differently.

Your cardinality is legitimate and bounded. If you have 200,000 series because you have 200,000 customers and you're intentionally tracking per-customer SLOs, that's not a cardinality problem — that's your data model. Mimir handles this fine at scale. Don't apply normalization rules that destroy the semantics of your data.

You're running Collector as a sidecar per pod. The OTTL transform processor holds state in memory for the duration of a batch window. In a sidecar deployment with many short-lived pods, the memory overhead per pod can exceed the benefit. In this topology, push the normalization into the SDK using the AttributesProcessor from the OTel SDK, or use a central Collector gateway tier for the heavy transformation work.

The Instrumentation Contract

The underlying issue is that most engineering teams have no explicit contract for what attributes are allowed on metrics. Developers instrument code, attributes end up as Prometheus labels, and nobody finds out until Mimir starts rejecting writes.

The fix is organizational as much as technical. Define an allowlist of metric attributes that are permitted to become Prometheus labels. Encode that allowlist in your Collector config. Make the Collector the enforcement point. Treat cardinality violations the same way you'd treat a schema migration without a review — as a change that requires explicit approval.

The OTel semantic conventions give you a starting point. service.name, service.version, k8s.namespace.name, k8s.deployment.name, http.method, http.route, http.status_code, db.system, rpc.service, rpc.method — these are stable, bounded, and useful. url.path, url.full, db.statement, http.url, enduser.id — these are unbounded and should never be Prometheus labels.

Write this down. Put it in your Collector config as comments. The next engineer who adds a new SDK attribute will thank you when the Collector silently drops it instead of silently exploding your Mimir.

What to Do This Week

If you're running OTel instrumentation and Mimir (or any Prometheus-compatible backend), run this query against your current data:

topk(10, count by (__name__) ({__name__=~".+"}))

If any metric has more than 50,000 active series, you have a cardinality problem. Pull a sample of that metric's label sets and look for attributes with numeric IDs, UUIDs, or full URL paths. Those are your starting points.

The Collector config patterns above are a starting point, not a complete solution — your specific attribute names will differ based on your SDK version and instrumentation library. The OTTL documentation for the transform processor is thorough and the replace_pattern function handles most normalization cases without custom code.

If you're in the middle of a migration from Datadog or Splunk and you're seeing cardinality numbers that don't make sense relative to your service count, we've seen this pattern enough times that we can usually identify the source within a single Collector pipeline review. Etalon offers a focused cardinality audit as part of migration engagements — it's usually the first thing we do before touching any storage configuration.