Scaling Grafana Alloy as a Telemetry Gateway on AWS: What the Load Tests Don't Tell You

Grafana Alloy's gateway mode looks straightforward on paper: run a small fleet of collectors in front of your Mimir, Loki, and Tempo backends, and stop punching thousands of individual agent connections directly at your storage tier. In practice, the jump from 'it works in staging' to 'it survives a Black Friday spike' involves a set of failure modes that capacity-planning spreadsheets miss entirely. We've deployed Alloy as a central gateway on AWS for three separate migrations this year. Here is what we learned that the official load-testing guide doesn't cover.

Why a Gateway Layer Exists (and Why It's Easy to Get Wrong)

Without a gateway, every application pod or EC2 instance with an OTel SDK opens a direct gRPC connection to your backend. At 500 pods that's 500 persistent connections to Mimir's distributor. At 5,000 pods — after a cluster autoscaler event during a traffic surge — that's 5,000. Mimir's distributor can handle fan-in, but each connection carries memory overhead, and the distributor's connection-handling goroutines become a bottleneck well before your ingest throughput limit.

Alloy as a gateway solves this by aggregating: your 5,000 pods talk to a small Alloy fleet (say, 6 instances behind an AWS Network Load Balancer), and those 6 instances maintain a stable, small connection pool to the backend. The fan-in ratio is the core value proposition.

The failure modes emerge from two places: the NLB configuration and Alloy's internal queue and retry behavior under backpressure.

The AWS NLB Misconfiguration That Will Ruin Your Day

gRPC runs over HTTP/2, which multiplexes multiple streams over a single TCP connection. An AWS Network Load Balancer operates at Layer 4 — it doesn't understand HTTP/2 streams. It load-balances at the TCP connection level.

This creates a silent imbalance problem. If your 5,000 application pods establish persistent gRPC connections and the NLB does connection-level hashing, you can end up with 80% of your streams landing on two Alloy instances while four others sit near-idle. We've seen this in production: CPU and memory metrics on the Alloy fleet look fine in aggregate, but two nodes are OOM-killed during a spike while the others have headroom.

The fix is to stop treating this as a standard TCP load balancing problem:

# Alloy receiver config — force HTTP/2 keepalive to encourage
# client-side reconnection and re-balancing
otelcol.receiver.otlp "gateway_receiver" {
  grpc {
    endpoint = "0.0.0.0:4317"
    keepalive {
      server_parameters {
        max_connection_age      = "30s"
        max_connection_age_grace = "5s"
        time                   = "10s"
        timeout                = "5s"
      }
    }
  }
}

Setting max_connection_age to 30 seconds forces clients to periodically reconnect. On reconnection, the NLB's connection hashing redistributes them. It's not elegant — you're working around a Layer 4 limitation with application-layer keepalive tuning — but it works. After applying this across our three deployments, peak CPU variance across the Alloy fleet dropped from ±60% to ±12%.

The alternative is to put an AWS Application Load Balancer in front with gRPC protocol support enabled, which gives you HTTP/2-aware routing. The tradeoff: ALB adds ~1-2ms of latency per request and costs more at high request rates. For most workloads the NLB + keepalive approach is the right call. For workloads where you're sending >500K spans/second through the gateway, benchmark both.

Queue Depth and the Backpressure Cascade

Alloy's pipeline has an internal queue between receivers and exporters. When the downstream backend (Mimir, Loki, Tempo) slows down — a compaction storm, a network hiccup, an S3 throttle — Alloy's exporter queue fills. Once the queue is full, Alloy applies backpressure to its receivers. The receivers then return gRPC errors to the sending agents.

This is correct behavior. The problem is the default queue configuration is sized for a single-node collector, not a gateway handling aggregated load.

otelcol.exporter.otlphttp "mimir_exporter" {
  client {
    endpoint = "http://mimir-distributor.monitoring.svc:8080/otlp"
    timeout  = "10s"
  }

  queue {
    enabled        = true
    num_consumers  = 20       # default is 10; double for gateway workloads
    queue_size     = 10000    # default is 1000; 10x for gateway workloads
  }

  retry {
    enabled          = true
    initial_interval = "1s"
    max_interval     = "30s"
    max_elapsed_time = "300s"
  }
}

The queue_size of 10,000 sounds large. At 100,000 spans/second per Alloy node, with each batch containing 1,000 spans, that's 100 batches/second. A 30-second backend hiccup generates 3,000 queued batches. 10,000 gives you roughly 100 seconds of buffer — enough to survive most transient issues without dropping data.

Memory cost: each queued batch is held in-heap. At 1,000 spans/batch with average span size of 2KB, 10,000 queued batches is ~20GB. That's why we run Alloy gateway nodes on r7g.2xlarge instances (64GB RAM) rather than the c7g.2xlarge (16GB RAM) you might reach for when optimizing for CPU.

Monitoring the Queue Itself

The queue depth is the leading indicator. If you're not alerting on it, you're flying blind.

# Alert when queue is more than 70% full on any gateway instance
(
  otelcol_exporter_queue_size
  /
  otelcol_exporter_queue_capacity
) > 0.7

Pair this with a dashboard panel showing otelcol_exporter_send_failed_metric_points_total rate. A rising queue depth with no failed sends means you're buffering successfully. A rising queue depth with rising failed sends means you're about to drop data and the backend problem needs immediate attention.

The S3 Throttling Problem Nobody Mentions

Mimir, Loki, and Tempo all use S3 as their object store on AWS. S3 has per-prefix request rate limits: 3,500 PUT/COPY/POST/DELETE requests/second and 5,500 GET/HEAD requests/second per prefix. This is per-prefix, not per-bucket, which is why the guidance is to use randomized or hash-distributed prefixes.

But here's what bites gateway deployments specifically: when Alloy is successfully buffering a backend hiccup and then the backend recovers, it flushes its queue in a burst. All six Alloy nodes flush simultaneously. That burst hits Mimir's distributor, which immediately fans out to ingesters, which immediately flush to S3. If your S3 prefix structure isn't well-distributed, you hit the per-prefix rate limit and get 503s from S3, which look to Mimir like a storage failure, which causes Mimir to apply backpressure, which refills Alloy's queue, which causes another burst when Mimir recovers. We've watched this oscillation loop for 20 minutes before catching it.

The fix is on the Mimir side, not the Alloy side:

# mimir.yaml — ensure blocks storage uses randomized prefix
blocks_storage:
  s3:
    bucket_name: your-mimir-blocks-bucket
    # Use a prefix that distributes across S3's internal partitioning
    # Empty prefix is fine if you have a dedicated bucket
    # Avoid a single static prefix like 'mimir/' across all tenants
  tsdb:
    block_ranges_period: [2h]
    retention_period: 0  # handled at compactor level

More practically: use a dedicated S3 bucket per signal type (one for Mimir blocks, one for Loki chunks, one for Tempo traces). Dedicated buckets eliminate cross-signal prefix competition and make IAM policies cleaner. The cost difference is negligible; S3 charges per request and per GB, not per bucket.

Capacity Planning Numbers That Actually Hold

Here are the numbers from our three production deployments. These are real, from AWS Cost Explorer and Grafana dashboards, not theoretical.

The rule of thumb that held across all three: 1 r7g.2xlarge per 50,000 peak spans/second, plus 20% headroom. For metrics, the limiting factor is usually network bandwidth before CPU — each r7g.2xlarge has up to 10 Gbps, and at 2.1M active series you're pushing roughly 3-4 Gbps of compressed protobuf during scrape cycles.

For log volume, Loki's gateway path is cheaper than Tempo's because log entries are typically smaller than spans and Loki's compression ratios are better. Budget 1 node per 5 GB/min of uncompressed log volume at peak.

When Not to Use a Centralized Gateway

This architecture is not always the right answer.

Don't use a centralized gateway if your application pods are in multiple AWS regions and your backends are single-region. You'd be routing telemetry from us-west-2 pods to a gateway in us-east-1, paying cross-region data transfer costs and adding 60-80ms of latency to every telemetry write. In this case, run regional Alloy gateway fleets that write to regional backends, or accept direct agent-to-backend connections within each region.

Don't use a centralized gateway if your total telemetry volume is low. Below roughly 10,000 spans/second and 500K active metric series, the operational overhead of maintaining a gateway fleet exceeds the benefit. Direct agent connections to Mimir/Loki/Tempo work fine at this scale.

Don't use a centralized gateway if you need per-tenant routing logic that changes frequently. Alloy's routing is configured in HCL and requires a reload or restart to change. If you need dynamic per-tenant routing — sending tenant A's data to a dedicated Loki instance while tenant B goes to a shared one — you need a more sophisticated control plane, or you need to accept that Alloy reloads are part of your operational workflow.

The Operational Runbook You Need Before Go-Live

Three things that should be in your runbook before you put production traffic through an Alloy gateway fleet:

1. Graceful drain on instance replacement. When you roll a new Alloy version or replace an instance, you need to drain in-flight batches before termination. Set your ASG lifecycle hook to call Alloy's admin API and wait for queue depth to reach zero before allowing the instance to terminate. Alloy exposes /-/ready and its queue metrics via the built-in Prometheus endpoint on port 12345.

2. Backend circuit breaking. If Tempo goes down completely, Alloy will retry indefinitely and fill its trace queue. Meanwhile, metrics and logs (which go to Mimir and Loki) continue fine. Make sure your alerting distinguishes "Alloy trace queue full" from "Alloy metric queue full" — they have different root causes and different remediation paths.

3. Load test with realistic burst profiles, not steady-state. The Grafana Labs load testing guidance uses steady-state throughput. Production traffic is not steady-state. Run your load test with a ramp from 0 to 3x expected peak over 60 seconds, hold for 5 minutes, then drop to 0. Watch what happens to queue depth during the ramp and during the drain. That's where the failure modes live.

Where to Go From Here

The Grafana Labs blog post on scaling Alloy as a telemetry gateway is a good starting point for understanding the architecture. What it doesn't cover — and what this post is about — is the AWS-specific failure modes and the operational details that only show up in production.

If you're planning a gateway deployment, the sequence we recommend is: (1) validate your NLB keepalive configuration in a staging environment with a connection-distribution test, (2) size your queue based on your expected backend recovery time not your steady-state throughput, (3) run a burst load test before any production cutover.

We've written the runbooks, hit the S3 throttling walls, and debugged the NLB connection imbalance more than once. If you're mid-migration and hitting any of these issues, or if you're planning a gateway deployment and want a second opinion on your architecture, we're at etalon.systems. We do architecture reviews as a fixed-scope engagement before any larger project — no commitment required.