Scaling Grafana Alloy as a Telemetry Gateway on AWS: What the Benchmarks Don't Tell You

Grafana Alloy looks like a straightforward drop-in for your telemetry pipeline. Run it as a DaemonSet or a central gateway, point your apps at it, done. That story holds until you're pushing 2 million spans per minute through a single gateway tier and watching your ECS tasks OOM-kill themselves at 3am.

We've run Alloy in production gateway mode on AWS across several migrations from Datadog and Splunk. The capacity planning story is more nuanced than the official benchmarks suggest — and the failure modes are specific enough that a generic "scale horizontally" answer is actually harmful.

This post covers what we've learned: where Alloy's memory model bites you, how to size an AWS gateway fleet, and the configuration knobs that actually matter.

Why Central Gateway Mode Instead of DaemonSet

The DaemonSet model is fine for homogeneous Kubernetes workloads. Every node runs an Alloy instance, traffic is local, no network hop. Simple.

Central gateway mode becomes the right answer in three situations we see repeatedly:

1. Mixed infrastructure. You have ECS Fargate tasks, Lambda functions, EC2 instances, and EKS pods all emitting telemetry. A DaemonSet only covers the EKS nodes. Everything else needs somewhere to send data.
2. Egress cost control. In a multi-AZ setup, crossing AZ boundaries costs $0.01/GB each way. If your DaemonSet agents are fanning out to backends in a different AZ, that adds up. A gateway per AZ with local agents pointing to it keeps cross-AZ traffic predictable.
3. Tail sampling. You cannot do probabilistic tail sampling at the agent layer because individual agents see only a fraction of spans for any given trace. Tail sampling requires a component that sees the full trace, which means a gateway that receives all spans before making a keep/drop decision.

If none of those three apply, stick with DaemonSet. The gateway adds latency (typically 2-8ms in our measurements, depending on batch flush intervals) and operational surface area. Don't pay that cost unnecessarily.

The Memory Model You Need to Understand Before Sizing

Alloy's OTEL receiver pipeline buffers data in memory between components. The relevant chain for a typical gateway looks like this:

OTLP receiver → batch processor → [optional: tail sampler] → OTLP exporter

Each arrow is a channel with a configurable queue depth. The batch processor holds spans in memory until either send_batch_size is reached or timeout fires. The exporter queue holds batches waiting to be sent to the backend (Tempo, Mimir, Loki).

The failure mode: if your backend is slow or temporarily unavailable, the exporter queue fills up. Once it's full, backpressure propagates upstream to the batch processor, which then holds more data in memory. If you've also enabled tail sampling, the tail sampler is holding complete traces in memory for the duration of decision_wait (commonly 30 seconds). These three buffers compound.

We had a client running Tempo on a single r6g.2xlarge (64GB RAM). During a Tempo compaction spike, write latency jumped from ~40ms to ~800ms for about 4 minutes. Their Alloy gateway fleet — sized for normal conditions — held 90 seconds of buffered trace data in memory per instance before the exporter queue started dropping. Three of five gateway instances hit their container memory limit and were killed by ECS before Tempo recovered.

The fix was not "add more memory." It was understanding which buffer to tune and accepting the tradeoff explicitly.

Sizing the Gateway Fleet on AWS

Here's the framework we use. It's not a formula — it's a set of questions that produce a sizing decision.

Step 1: Measure your actual telemetry volume

Before touching Alloy configuration, instrument the pipeline itself. Alloy exposes Prometheus metrics. The ones you need:

# Spans received per second at the OTLP receiver
rate(otelcol_receiver_accepted_spans_total[1m])

# Spans dropped by the batch processor (queue full)
rate(otelcol_processor_dropped_spans_total[1m])

# Exporter queue depth (0.0 to 1.0)
otelcol_exporter_queue_size / otelcol_exporter_queue_capacity

# Memory in use by the Alloy process
process_resident_memory_bytes

If you don't have these dashboards yet, build them before you size anything. You need p95 and p99 of spans/sec over a 2-week window, not the average. Traffic is spiky.

Step 2: Establish your baseline memory budget per instance

For a gateway without tail sampling, the memory consumed by in-flight data is approximately:

memory_for_buffers = (spans_per_second × avg_span_size_bytes × buffer_window_seconds)

Where buffer_window_seconds is the sum of your batch timeout plus the time it takes to flush the exporter queue under normal backend latency.

For a typical microservices workload, average span size after OTLP encoding is 800-1200 bytes. If you're sending 50,000 spans/sec through a gateway instance with a 5-second batch timeout and 2-second flush time:

50,000 × 1,000 bytes × 7 seconds = 350MB

That's just the telemetry buffers. Add Alloy's base process overhead (~150MB on a warmed-up instance), Go runtime overhead, and a 40% headroom buffer for traffic spikes. You're at ~650MB minimum per instance for this workload.

With tail sampling at decision_wait = 30s, the math changes completely:

50,000 spans/sec × 1,000 bytes × 30 seconds = 1.5GB just for the trace buffer

Tail sampling is expensive. Size for it explicitly or don't enable it.

Step 3: Choose instance type and count

We run Alloy gateways on c7g.xlarge (4 vCPU, 8GB RAM, Graviton3) in ECS Fargate or on EC2 with an Auto Scaling Group. Alloy is CPU-bound at high throughput (OTLP decoding, protobuf marshaling, compression), not memory-bound under normal conditions. Graviton3 gives better price-performance for CPU-intensive Go workloads than equivalent x86 options in our benchmarks.

These numbers are from our load tests using telemetrygen generating realistic OTLP payloads against Alloy 1.3.x, exporting to Tempo on the same VPC. Your numbers will differ based on span complexity, compression settings, and backend latency. Treat these as order-of-magnitude starting points, not guarantees.

For a fleet, we recommend a minimum of 3 instances behind an NLB (Network Load Balancer, not ALB — OTLP/gRPC needs HTTP/2, and ALB's HTTP/2 support adds latency that NLB avoids). Size for 60% utilization at p95 traffic, not average. The headroom is not waste; it's what keeps you alive during a backend hiccup.

The Configuration That Actually Matters

Most Alloy gateway configuration guides show you the minimum viable config. Here's what we actually run in production:

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
    max_recv_msg_size_mib = 32
  }
  http {
    endpoint = "0.0.0.0:4318"
  }
  output {
    traces  = [otelcol.processor.batch.default.input]
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  send_batch_size     = 8192
  send_batch_max_size = 16384
  timeout             = "5s"

  output {
    traces  = [otelcol.exporter.otlphttp.tempo.input]
    metrics = [otelcol.exporter.otlphttp.mimir.input]
    logs    = [otelcol.exporter.otlphttp.loki.input]
  }
}

otelcol.exporter.otlphttp "tempo" {
  client {
    endpoint = "http://tempo-distributor.monitoring.svc:4318"
    compression = "zstd"

    sending_queue {
      enabled        = true
      num_consumers  = 8
      queue_size     = 2000
    }

    retry_on_failure {
      enabled          = true
      initial_interval = "1s"
      max_interval     = "30s"
      max_elapsed_time = "5m"
    }
  }
}

Three things worth explaining:

send_batch_max_size = 16384: Without this, a single slow flush can accumulate an unbounded batch. Setting send_batch_max_size to 2x send_batch_size caps the maximum memory a single batch can consume.

num_consumers = 8: This controls the number of goroutines draining the exporter queue. Default is 10, which is fine for most cases, but if you're seeing queue buildup under load, increasing this helps — up to the point where you're saturating the backend connection pool. We tune this per backend.

compression = "zstd": zstd at default compression level reduces OTLP payload size by 60-75% versus uncompressed, with CPU cost that's negligible on Graviton3. This matters for egress costs when your gateway is sending to a backend in a different AZ or region.

Where This Approach Fails

I want to be direct about the cases where central gateway mode on Alloy is not the right answer.

Very high cardinality metrics with exemplars. If you're emitting histograms with exemplars at high cardinality (millions of unique label combinations), the Alloy metrics pipeline can become a bottleneck. The batch processor does not deduplicate or aggregate — it just batches. Aggregation needs to happen either at the SDK level or in a dedicated metrics pipeline (Prometheus remote write with recording rules). We've seen Alloy instances spend 40% of CPU on metrics serialization when exemplar cardinality is pathological.

Latency-critical traces below 10ms SLA. The gateway adds a minimum of one batch timeout to your trace delivery latency. With a 5-second batch timeout, your spans don't reach Tempo for up to 5 seconds after emission. For most use cases this doesn't matter — you're querying Tempo minutes after the fact. But if you're building real-time alerting on span data, this latency is a constraint you need to design around.

Small teams without dedicated platform engineering capacity. Running a gateway fleet means operating another distributed system: NLB health checks, Auto Scaling policies, Alloy version upgrades, configuration management. If your team is three people and you're already stretched, the operational overhead of a gateway fleet may not be worth it versus paying for Grafana Cloud's managed ingest tier. The cost math changes when you factor in engineering time honestly.

Load Testing Before You Go Live

Do not go to production without a load test. The Grafana Labs post on this topic is worth reading, but here's our minimal viable test setup:

# Install telemetrygen
go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest

# Generate 50,000 spans/sec for 10 minutes against your gateway
telemetrygen traces \
  --otlp-endpoint your-gateway-nlb:4317 \
  --otlp-insecure \
  --rate 50000 \
  --duration 10m \
  --workers 20 \
  --telemetry-attributes "service.name=load-test,env=staging"

Watch three things during the test: 1. otelcol_exporter_queue_size / otelcol_exporter_queue_capacity — if this approaches 1.0, you're about to drop data 2. process_resident_memory_bytes — watch for growth that doesn't plateau, which indicates a leak or unbounded buffer accumulation 3. ECS/EC2 CPU utilization — if you're above 70% at your target load, you don't have headroom for spikes

Then deliberately degrade your backend: throttle Tempo's ingest endpoint with a simple iptables rule or a chaos engineering tool. Watch how the gateway behaves under backpressure. This is the test that reveals your actual failure mode, not the happy-path load test.

What a Production-Ready Gateway Architecture Looks Like on AWS

For a fleet handling 100,000-500,000 spans/second:

• NLB (Network Load Balancer) in each AZ, with cross-zone load balancing disabled. Keep traffic local to the AZ where it originates.
• 3-5 c7g.2xlarge instances per AZ in an Auto Scaling Group, scaling on otelcol_exporter_queue_size (custom CloudWatch metric via the Alloy Prometheus scrape + CloudWatch metric stream) rather than CPU. Queue depth is a leading indicator of saturation; CPU is a lagging one.
• Alloy config managed via AWS AppConfig or SSM Parameter Store, with a rolling restart on config change. Alloy 1.3+ supports config reload without restart for most components, but we still prefer a controlled rolling restart for config changes that touch the receiver or exporter.
• Separate gateway clusters per signal type (traces, metrics, logs) at high scale. Mixing signals in a single gateway is fine up to ~200,000 events/sec total. Above that, a metrics spike can starve the trace pipeline's goroutines. Separation gives you independent scaling and blast radius control.

Where to Go Next

The Grafana Labs post on scaling Alloy is a good starting point for understanding the load testing methodology. Their numbers are from a controlled environment; yours will differ based on span complexity, backend topology, and traffic patterns.

If you're in the middle of a migration from Datadog or Splunk and are trying to figure out whether your existing infrastructure can absorb the telemetry volume that you're currently paying a SaaS vendor to handle, the answer is almost always yes — but the sizing work has to be done carefully. We've seen teams under-provision by 3x and over-provision by 5x on their first attempt.

At Etalon, this is the work we do before any migration goes live: load test the target stack against production-representative traffic, find the failure modes, and fix them before the cutover date. If you're planning a migration and want a second opinion on your gateway architecture, we're reachable at etalon.systems.