OpenTelemetry Go Compile-Time Instrumentation v1: What It Actually Changes for Your AWS Services

The OpenTelemetry project shipped v1 of Go compile-time instrumentation this month. If you run Go services on AWS and you've been living with the manual span-annotation tax — or worse, skipping instrumentation entirely because the setup cost wasn't worth it — this release changes the math. Here's what it does, where it falls short, and how we've been integrating it into Grafana/Tempo pipelines for clients migrating off Datadog APM.

Why Go Instrumentation Was Always Painful

Every other major runtime has a story for automatic instrumentation. Java has the agent JAR. Python has opentelemetry-instrument. Node has --require @opentelemetry/auto-instrumentations-node. You attach something at startup and you get traces for free.

Go never had that. Go compiles to a static binary. There's no JVM to hook, no interpreter to monkey-patch. The only options before this release were:

1. Manual instrumentation — wrap every http.Handler, every database call, every outbound gRPC stub by hand. Correct, but the diff to add tracing to a mature service is often 300-500 lines across dozens of files.
2. eBPF-based auto-instrumentation — tools like Odigos or Grafana Beyla attach at the kernel level. Works without code changes, but adds operational complexity, requires privileged containers or a DaemonSet, and the coverage is limited to network boundaries. You get entry/exit spans for HTTP and gRPC; you don't get internal spans.
3. Accept the gap — many teams just don't instrument internal Go services deeply. They get metrics from Prometheus and logs from Loki and call it good enough.

Option 3 is more common than anyone admits. When we run an observability audit before a Datadog migration, we consistently find that Go services have 40-60% less trace coverage than Java or Python equivalents in the same codebase.

What Compile-Time Instrumentation Actually Does

The v1 release ships a tool called otel-go-build (the project calls it the "compile-time instrumentation agent"). You add it to your build pipeline. It rewrites your Go AST at compile time to inject OTel spans into supported libraries before the compiler sees the final code.

The practical effect: you run:

# Before: standard build
go build -o myservice ./cmd/myservice

# After: instrumented build
otel-go-build -o myservice ./cmd/myservice

And your binary now emits spans for net/http, database/sql, google.golang.org/grpc, github.com/go-redis/redis, and a growing list of other packages — without a single line change in your application code.

The AST rewriting approach is not magic. It's a well-understood pattern. The tool injects wrapper code around function calls to instrumented libraries. Because this happens at compile time, there's no runtime reflection, no interface boxing overhead, and the resulting binary is a normal Go binary — no agent process, no sidecar.

What the Injected Code Looks Like

For a simple http.Get call, the transformation is roughly equivalent to this (you don't write this; the tool injects it):

// Original code in your service:
resp, err := http.Get("https://api.internal/data")

// What the compiled binary behaves as after instrumentation:
ctx, span := tracer.Start(ctx, "HTTP GET api.internal")
resp, err := otelhttp.Get(ctx, "https://api.internal/data")
span.SetAttributes(
    semconv.HTTPMethodKey.String("GET"),
    semconv.HTTPURLKey.String("https://api.internal/data"),
    semconv.HTTPStatusCodeKey.Int(resp.StatusCode),
)
span.End()

The semantic conventions are baked in. The spans are correctly parented to whatever context exists in the goroutine. If there's no parent context, the tool creates a root span.

Integrating This Into a Grafana/Tempo Pipeline on AWS

For clients on ECS Fargate or EKS, the integration path looks like this:

Build Pipeline Changes

You add otel-go-build to your Docker build stage. A typical multi-stage Dockerfile addition:

FROM golang:1.24 AS builder

# Install the instrumentation tool
RUN go install go.opentelemetry.io/contrib/instrgen/cmd/otel-go-build@v1.0.0

WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .

# Replace 'go build' with 'otel-go-build'
RUN otel-go-build -o /app/bin/myservice ./cmd/myservice

FROM gcr.io/distroless/static-debian12
COPY --from=builder /app/bin/myservice /myservice
ENTRYPOINT ["/myservice"]

Build time increase in our testing on a mid-sized service (~80k lines of Go): approximately 18 seconds on a 4-core build runner. Acceptable.

OTel Collector Configuration

The instrumented binary exports via OTLP. We route through an OTel Collector sidecar (on ECS) or DaemonSet (on EKS) to Tempo. A minimal collector config that works:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  resource:
    attributes:
      - key: deployment.environment
        value: ${ENVIRONMENT}
        action: upsert
      - key: cloud.provider
        value: aws
        action: insert
      - key: cloud.region
        value: ${AWS_REGION}
        action: insert

exporters:
  otlp:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true  # TLS termination handled at the ingress

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlp]

The resource processor is doing real work here. When you migrate from Datadog, your traces lose the Datadog agent's automatic AWS metadata injection. You need to explicitly attach cloud.region, deployment.environment, and your service name. Skipping this makes cross-service correlation in Tempo significantly harder.

Environment Variables for the Instrumented Binary

# Required
OTEL_SERVICE_NAME=payment-service
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# Strongly recommended
OTEL_RESOURCE_ATTRIBUTES=service.version=${IMAGE_TAG},team=payments
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1  # 10% head sampling; adjust per service volume

The sampling rate deserves attention. At 100% sampling on a high-throughput Go service (say, 5,000 RPS), you will generate more trace data than you expect. On one recent engagement, a client's payment service at 100% sampling produced 2.8 TB/day of trace data. At 10% head sampling with tail sampling configured in the collector for errors and slow requests, that dropped to 180 GB/day with no meaningful loss of diagnostic coverage.

What You Actually Get in Tempo

After this setup, in Grafana's Tempo datasource you get:

• Full distributed traces across service boundaries with correct parent-child relationships
• Automatic span attributes following OTel semantic conventions (HTTP method, status code, DB statement, gRPC status)
• Trace-to-log correlation when you configure the OTel logging bridge alongside the tracer
• Service graph in Grafana derived from span relationships — no separate topology agent needed

Here's a comparison of what we typically see before and after on a Go microservices migration:

The span count jump from 12 to 31 is because the manual instrumentation had gaps — developers instrumented the obvious paths and skipped internal DB calls and Redis lookups. The compile-time tool catches everything the supported library list covers.

The trace query latency improvement is a Tempo vs. Datadog APM comparison on the same dataset size. Tempo's columnar storage in object storage (S3 in this case) with local disk caching on the query frontend is genuinely fast for the trace-by-ID and service-graph queries that dominate interactive debugging.

Where This Approach Falls Short

I want to be direct about the limitations, because there are real ones.

Library coverage is not complete. The v1 release covers the most common packages, but if you use less mainstream libraries — certain AWS SDK v2 call patterns, custom TCP protocols, or niche database drivers — you will not get automatic spans for those. You still need manual instrumentation for those paths. Check the supported packages list before assuming full coverage.

Context propagation in goroutines is still your problem. If your code spawns goroutines and you don't pass context correctly, the compile-time tool cannot fix that. Spans will be disconnected. The tool injects instrumentation at call sites, but it cannot infer which goroutine belongs to which request if you haven't threaded context through. This is a Go language constraint, not a tool limitation.

Build reproducibility requires version pinning. The tool rewrites your AST. If otel-go-build version changes between builds, your binary's instrumentation behavior can change. Pin the version in your Dockerfile and treat upgrades as intentional changes. We've seen teams get surprised by span attribute name changes between minor versions.

This is not a replacement for Beyla if you can't modify builds. If you're running third-party Go binaries, or if your build pipeline is locked down and you can't add a build step, eBPF-based instrumentation (Grafana Beyla, specifically) is still the right tool. Compile-time instrumentation requires owning the build.

The v1 label is real but the ecosystem is young. v1 means the API is stable. It does not mean the tool has been battle-tested at the scale of a Datadog agent with years of production hardening. Run it in staging for a few weeks before rolling to production, and have a fallback build target without instrumentation ready.

The Migration Sequence We Use

For clients moving Go services from Datadog APM to Tempo, our current recommended sequence:

1. Deploy Tempo and OTel Collector in your AWS environment first. Get the pipeline working with manually-instrumented test spans before touching production services.
2. Start with one low-risk Go service. Add otel-go-build to its Dockerfile. Deploy to staging. Validate spans appear in Tempo with correct attributes and parent-child relationships.
3. Enable trace-to-log correlation by configuring the OTel log bridge alongside the tracer. This is the feature that usually impresses engineers most during the migration — clicking from a slow span directly to the relevant log lines in Loki without copy-pasting a trace ID.
4. Set sampling rates deliberately. Don't run 100% in production until you've measured the data volume. Use tail sampling in the collector to keep errors and slow requests at 100% while sampling normal traffic at 10-20%.
5. Run Datadog and OTel in parallel for 2-4 weeks. Compare coverage. When you're confident OTel coverage is equal or better, cut over.
6. Remove Datadog agent. Not before step 5 is complete.

The parallel run period is non-negotiable in our engagements. The cost of running both for a month is small. The cost of cutting over and discovering a coverage gap during an incident is not.

What This Means for Datadog Migration Economics

The historical argument against migrating Go-heavy stacks off Datadog APM was engineering cost. Manual OTel instrumentation of 15 Go services was a 3-6 month project. That labor cost ate into the SaaS savings.

Compile-time instrumentation changes that calculation. On a recent engagement with a fintech running 18 Go services, we estimated the instrumentation work at 6 weeks of engineering time under the old approach. With otel-go-build handling the library-level spans, that dropped to about 10 days — mostly configuration, sampling tuning, and validating coverage on the non-standard library paths that still needed manual spans.

The annual Datadog APM bill for those 18 services was $340,000. The equivalent Tempo + Grafana Cloud (for the UI only, traces stored in their own S3) came to $31,000. The instrumentation labor cost was recovered in under 6 weeks.

Those numbers are from one specific client. Your numbers will differ based on ingestion volume, retention requirements, and whether you self-host Grafana or use Grafana Cloud. I'm not going to tell you the ROI is always the same, because it isn't.

Where to Go From Here

If you're evaluating this for your stack:

• The OTel Go compile-time instrumentation v1 announcement has the full supported library list and migration notes.
• Grafana's Tempo documentation covers the trace storage configuration, including the S3 backend setup and query frontend caching that drives the query performance numbers above.
• Grafana Beyla is still the right choice if you can't modify builds — the two tools are complementary, not competing.

If you're running Go services on AWS and spending meaningfully on Datadog APM, the instrumentation cost objection to migrating is now substantially weaker than it was six months ago. Whether the total migration is the right move for your organization depends on your team's operational appetite for running the stack, not on instrumentation effort.

We've done this migration enough times to have opinions about where it goes wrong. If you want a frank conversation about whether your specific stack is a good candidate, Etalon's discovery call is a 45-minute technical conversation, not a sales pitch.