OTel-Arrow Phase 2 in Production: What the Compression Numbers Actually Mean for Your Loki and Mimir Bills

OpenTelemetry Arrow Phase 2 shipped last month and the project blog claims 70–90% wire-size reduction for telemetry data. That number is real — but it applies narrowly, and the path from 'efficient transport' to 'lower AWS bill' has several steps that the announcement glosses over.

Here is what we have found running OTel-Arrow in pre-production against Loki 3.x and Mimir 2.x on AWS, and where the gains actually land.

What OTel-Arrow Phase 2 Actually Changed

Phase 1 introduced Arrow-encoded OTLP as an experimental exporter in the OpenTelemetry Collector. It replaced the default Protobuf-over-gRPC encoding with Apache Arrow IPC format, which columnar-packs repeated fields — timestamps, attribute keys, service names — that are highly redundant across spans and metrics.

Phase 2 extended this in two directions:

1. Pipeline-aware batching. The Arrow exporter can now hold records across multiple ResourceSpans or ResourceMetrics batches and re-sort them by schema before encoding. This is where the big compression wins come from: if your spans from the same service arrive in bursts across multiple export cycles, Phase 2 can coalesce them into a single Arrow record batch before sending.

2. Receiver-side acknowledgment. Phase 2 added a proper flow-control handshake so the collector receiver can apply backpressure without dropping data silently. This matters more for reliability than for compression, but it removes the main reason teams avoided Arrow in high-throughput paths.

The 70–90% wire-size reduction the project cites is measured on homogeneous trace data — a single service emitting spans with a stable attribute schema. Real production traffic is messier.

The Compression Reality Check

We ran OTel-Arrow Phase 2 against three representative workloads from recent client environments. All tests used otelcol-contrib 0.104.0, Arrow exporter with zstd compression, batching at 8,192 spans per flush, sending to an OTel Collector gateway running on m7i.2xlarge EC2 instances.

The 71% case is real and reproducible. The 19% case on logs is also real, and it is the one that stings — because logs are typically the largest signal by volume in the stacks we see, and the free-text Body field does not compress well in columnar layout. zstd still helps, but Arrow's columnar advantage disappears when the high-cardinality column is the one carrying most of the bytes.

Where This Shows Up (and Doesn't) in Your AWS Bill

Let's be specific about what 'wire reduction' translates to in a typical AWS-hosted Grafana stack.

Data Transfer Costs

If your instrumented services and your OTel Collector gateway are in the same VPC, intra-AZ traffic is free and inter-AZ traffic costs $0.01/GB. At 1 TB/day of trace data — a large but not unusual number for a 50-service platform — inter-AZ transfer is around $10/day. A 50% reduction saves $5/day. That is $1,825/year. Not nothing, but not the headline number.

If you are sending telemetry across regions or to a centralized observability account over a Transit Gateway or VPC peering, the math changes. Transit Gateway costs $0.02/GB processed plus $0.02/GB attachment-hour. Cross-region is $0.09/GB. At 1 TB/day cross-region, you are spending $90/day on transfer alone. A 50% reduction there is $16,000/year. That is worth engineering time.

Collector CPU and Memory

Arrow encoding is CPU-intensive on the exporter side. The columnar sort and IPC serialization at 8,192-span batches adds roughly 15–25% CPU overhead on the collector process compared to standard OTLP/proto. On a c7i.xlarge (4 vCPU, 8 GB), we saw the Arrow exporter saturate at around 180k spans/second where the proto exporter handled 230k spans/second at the same batch size.

This matters if you are running collectors as a DaemonSet on your EKS nodes — you are trading compute for bandwidth. On Graviton3 (c7g family), the gap narrows to about 10% because the NEON SIMD path in the Arrow C++ library is well-optimized for ARM. If you are already on Graviton for cost reasons, Arrow becomes more attractive.

Storage: The Leg That Doesn't Bend

This is the part the announcement does not emphasize enough: OTel-Arrow compression ends at the receiver. Loki, Mimir, and Tempo decompress the Arrow payload and store data in their own formats — Loki chunks (snappy or zstd), Mimir TSDB blocks, Tempo Parquet. The wire compression does not reduce your S3 storage costs.

For Datadog and Splunk migrations specifically, this is important context. Clients often expect that switching to an Arrow-based pipeline will reduce their object storage spend. It does not, directly. What reduces storage spend is Loki's chunk compression settings, Mimir's compaction configuration, and Tempo's TraceQL-driven retention policies — none of which are affected by the transport encoding.

The Architecture That Makes Arrow Worth It

The configuration that justifies Arrow in a production stack looks like this:

[Instrumented services]
      |
      | OTLP/gRPC (standard proto, intra-node)
      v
[OTel Collector DaemonSet on each EKS node]
      |
      | OTel-Arrow / gRPC + zstd (cross-AZ or cross-region)
      v
[OTel Collector Gateway — centralized, m7g.2xlarge]
      |
      | OTLP/HTTP (standard, intra-VPC)
      v
[Loki / Mimir / Tempo]

The Arrow leg is specifically the DaemonSet-to-Gateway hop. That is where you have the most to gain: high volume, repeated schemas per node, and a network boundary that costs real money.

Do not run Arrow end-to-end from application SDK to storage. The SDKs do not support it yet (as of June 2026, only the Collector exporters and receivers implement Phase 2), and even if they did, the per-process cardinality would be too low to benefit from columnar batching.

Collector Gateway Configuration

This is a working otelcol-contrib configuration for the gateway receiver side:

receivers:
  otelarrow:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
    arrow:
      memory_limit_mib: 512
      # Admit up to 8 concurrent Arrow streams per receiver goroutine
      # before applying backpressure. Default is 4; increase for high fan-in.
      admission_limit_mib: 256

processors:
  batch:
    send_batch_size: 4096
    timeout: 2s
  memory_limiter:
    check_interval: 1s
    limit_mib: 3072
    spike_limit_mib: 512

exporters:
  otlphttp/loki:
    endpoint: http://loki-gateway.monitoring.svc.cluster.local:3100/otlp
  otlphttp/tempo:
    endpoint: http://tempo-distributor.monitoring.svc.cluster.local:4318
  prometheusremotewrite/mimir:
    endpoint: http://mimir-nginx.monitoring.svc.cluster.local:9009/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otelarrow]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otelarrow]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite/mimir]
    logs:
      receivers: [otelarrow]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/loki]

The admission_limit_mib setting is new in Phase 2 and is the backpressure knob. If you set it too low on a high-fan-in gateway (many DaemonSet nodes sending simultaneously), you will see ResourceExhausted errors on the exporter side. Set it too high and you risk OOM on a traffic spike. We start at 256 MiB and tune from there based on otelcol_receiver_refused_spans in Mimir.

When Arrow Is Not the Right Call

I want to be direct about three situations where we would not recommend Arrow Phase 2 today:

1. Low-volume environments. If you are ingesting fewer than 50k spans/minute, the batching overhead and operational complexity of Arrow do not pay off. Standard OTLP/gRPC with gzip is simpler and the savings are marginal.

2. Logs-heavy workloads without schema discipline. If your log Body fields are unstructured strings — which is most logs in practice — Arrow's columnar advantage is minimal and you are paying CPU for little gain. Use Loki's built-in push compression instead and invest the engineering time in structured logging at the source.

3. Teams without Collector operational experience. Arrow Phase 2 adds a new failure mode: Arrow stream negotiation failures that fall back silently to proto if misconfigured. We have seen teams run Arrow receivers for weeks while the exporters were actually sending proto because the otelarrow extension was not enabled on both sides. You need collector metrics dashboards that show the active encoding per pipeline before you trust Arrow is working.

What to Measure Before and After

If you are evaluating Arrow for your stack, measure these four things:

1. otelcol_exporter_sent_spans vs. otelcol_exporter_queue_size — baseline throughput and backpressure signal.
2. VPC Flow Logs bytes by source/destination — the ground truth for network cost reduction. Do not trust collector-reported metrics alone.
3. Collector pod CPU utilization (p95) — Arrow encoding cost shows up here.
4. otelcol_receiver_refused_spans / _metrics / _log_records — backpressure rejections, your canary for misconfigured admission limits.

If you are migrating from Datadog, you can use the Datadog Agent's existing OTLP ingestion path to forward to an Arrow-enabled collector without re-instrumenting your services. That is a useful transitional architecture: Datadog Agent → OTel Collector (Arrow) → Mimir/Loki/Tempo. It lets you validate the pipeline and measure real compression ratios against your actual traffic before you commit to full re-instrumentation.

The Honest Bottom Line

OTel-Arrow Phase 2 is a genuine improvement to the OTel transport layer. The compression numbers are real in the right conditions. For a platform team running a multi-region, multi-service stack on AWS with meaningful cross-AZ or cross-region telemetry volume, it is worth the engineering investment to enable it on the DaemonSet-to-Gateway leg.

It is not a storage cost reduction. It is not a silver bullet for log volume. And it requires more operational care than standard OTLP because the failure modes are less obvious.

For most of the migrations we run — from Datadog or Splunk to a self-hosted Grafana stack — the bigger cost levers are Loki chunk compression tuning, Mimir compaction block size, and Tempo's retention policies. Arrow is worth enabling once those are dialed in, not before.

If you are mid-migration or evaluating whether your current OTel Collector architecture is leaving money on the table, we are happy to do a 30-minute architecture review. No slides, just your config and our notes. Reach out at etalon.systems.