OTel Go Compile-Time Instrumentation v1 Is Production-Ready: What It Means for Your Self-Hosted Stack

The OpenTelemetry Go compile-time instrumentation library hit v1 in late July 2026. For teams running self-hosted Grafana stacks on AWS, this changes the calculus on Go service instrumentation in ways that aren't obvious from the release notes alone.

For the past three years, instrumenting Go services with OpenTelemetry has required one of two compromises: either you accept manual SDK instrumentation (which means touching every library call, every HTTP handler, every database client), or you accept the instability and runtime overhead of eBPF-based auto-instrumentation. The compile-time approach — using Go's //go:linkname and AST rewriting at build time — has been in experimental status long enough that most teams shelved it. v1 changes that.

This post covers what the v1 release actually delivers, how it fits into a self-hosted OTel Collector → Tempo → Grafana pipeline on AWS, and where it still falls short.

What Compile-Time Instrumentation Actually Does

The standard OTel Go SDK requires you to explicitly wrap library calls:

// Manual instrumentation — you write this for every handler
func handler(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "handler")
    defer span.End()
    // ... your logic
}

Compile-time instrumentation rewrites your binary at build time using a custom toolchain wrapper (otel-go-build). You run it instead of go build:

# Before: go build -o myservice ./cmd/myservice
# After:
otel-go-build -o myservice ./cmd/myservice

The tool injects trace context propagation, span creation, and attribute collection into supported libraries — currently net/http, database/sql, google.golang.org/grpc, go.mongodb.org/mongo-driver, and a handful of AWS SDK v2 clients — without you modifying a single line of application code.

The mechanism is AST rewriting, not eBPF. It reads your source, identifies call sites for instrumented packages, and emits modified source that gets compiled normally. The output is a standard Go binary. No kernel privileges required at runtime. No agent sidecar. No LD_PRELOAD tricks.

Why This Matters for Self-Hosted Stacks Specifically

If you're on Datadog or New Relic, their agents handle this for you and you don't think about it. When you migrate off those platforms, the instrumentation gap is one of the first things that bites you.

We've migrated several Go-heavy platforms from Datadog to self-hosted Grafana stacks. In every case, the initial trace coverage after migration was worse than what Datadog provided, because Datadog's agent was doing a lot of silent work that teams didn't realize they were depending on. The gap shows up immediately in Tempo: you have spans for the endpoints you manually instrumented, and silence everywhere else.

Compile-time instrumentation at v1 quality closes most of that gap for services that don't need custom span attributes on internal business logic. For a typical CRUD API backed by PostgreSQL over database/sql and exposed over net/http, you get:

• Inbound HTTP spans with method, route, status code
• Outbound SQL spans with query text (configurable redaction), duration, error
• Automatic W3C trace context propagation on outbound HTTP calls
• Service graph edges in Tempo/Grafana without any manual work

That's roughly 80% of what teams actually query in Tempo day-to-day.

Wiring It Into an AWS Self-Hosted Pipeline

Here's the architecture we use for Go services on ECS Fargate with compile-time instrumentation:

[Go service binary (otel-go-build)]
    |
    | OTLP/gRPC (localhost:4317)
    v
[OTel Collector sidecar container]
    |
    |-- traces --> Tempo (ALB, port 4317, gRPC)
    |-- metrics --> Mimir (remote_write, port 9090)
    |-- logs --> Loki (via filelog receiver on /dev/stdout)

The Collector sidecar config for the traces path:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  resource:
    attributes:
      - key: deployment.environment
        value: "${DEPLOY_ENV}"
        action: upsert
      - key: service.version
        value: "${IMAGE_TAG}"
        action: upsert
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-policy
        type: latency
        latency: {threshold_ms: 500}
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

exporters:
  otlp:
    endpoint: tempo.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/internal-ca.crt

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch, tail_sampling]
      exporters: [otlp]

One thing worth calling out: the tail_sampling processor here is doing real work. Without it, a high-throughput Go service will flood Tempo with traces for successful 200ms health checks you will never look at. With the policy above, you capture 100% of errors, 100% of requests over 500ms, and 5% of everything else. On a service doing 2,000 RPS, that's the difference between 6TB/month and roughly 400GB/month in Tempo object storage on S3.

The Tempo backend config that pairs with this:

# tempo.yaml (relevant sections)
storage:
  trace:
    backend: s3
    s3:
      bucket: your-tempo-traces-bucket
      region: eu-west-1
      # Use instance profile — don't put credentials here
    pool:
      max_workers: 100
      queue_depth: 10000

compactor:
  compaction:
    block_retention: 720h  # 30 days
    compacted_block_retention: 1h

query_frontend:
  search:
    max_duration: 168h  # 7-day search window
    query_ingest_until: 30s

Honest Tradeoffs

Compile-time instrumentation v1 is not the right approach in every situation. Here's where it breaks down.

Build pipeline complexity. You now have a non-standard build step. Your Dockerfiles, CI pipelines, and any toolchain pinning need to account for otel-go-build. If you use Bazel or a heavily customized build system, the integration story is not smooth. The tool assumes a relatively standard go build invocation.

Limited library coverage. The supported library list at v1 is reasonable but not exhaustive. If your services use jackc/pgx directly instead of database/sql, you get nothing. If you use a custom gRPC interceptor framework, you get nothing. The project's roadmap lists jackc/pgx, go-redis/redis, and elastic/go-elasticsearch as near-term targets, but near-term in OTel project time has historically meant 6-18 months.

No custom span attributes on business logic. This is the fundamental limit of the approach. If you need a span attribute like order.customer_id or payment.method on a specific code path, you still need the SDK. Compile-time instrumentation gives you infrastructure-level spans. Application-level semantics require manual work. For many teams, that's fine — the infrastructure spans are what you need for latency debugging and error tracing. For teams doing detailed product analytics via traces, it's not enough.

Go version coupling. The tool currently supports Go 1.22 and 1.23. If you're on an older version for any reason (CGo constraints, vendor lock-in, internal policy), you're blocked. Go 1.21 support is listed as a known gap.

Reproducible builds. The AST rewriting means your compiled binary differs from what go build alone would produce. If you have strict reproducible-build requirements for compliance or supply chain security, you need to verify that the rewriting is deterministic (it is, per the v1 release notes) and document the tool in your SBOM pipeline.

Comparison: Compile-Time vs. eBPF vs. Manual SDK

The honest read: for greenfield Go services on Go 1.22+ that use standard library clients, compile-time instrumentation is now the right default. For existing services with heavy custom instrumentation already in place, the migration cost isn't worth it — keep the SDK. For services that need deep application-level tracing, use the SDK from the start.

What This Means for a Datadog Migration

When we scope a Datadog-to-Grafana migration for a Go-heavy platform, instrumentation coverage is always one of the first risk items. The question is: how much of what Datadog silently provided will we need to rebuild manually?

With compile-time instrumentation at v1, the answer for standard Go services is: not much. The coverage for net/http + database/sql + gRPC is solid enough that you can instrument a typical microservice fleet without touching application code. That's a meaningful reduction in migration scope.

The remaining gap is usually: 1. Services using non-covered libraries (pgx, Redis, Elasticsearch) 2. Services where teams built custom Datadog APM integrations using the DD SDK directly 3. Services where trace-based alerting depends on custom span attributes

For (1), the workaround is thin SDK wrappers around the library client — a one-time cost per library, not per service. For (2) and (3), manual SDK instrumentation is unavoidable, but it's scoped to specific services rather than the whole fleet.

Getting Started

If you want to evaluate compile-time instrumentation against your own services before committing to it, the fastest path is:

# Install the build tool
go install go.opentelemetry.io/auto/cmd/otel-go-build@v1

# Set a local OTel Collector endpoint for testing
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_SERVICE_NAME=my-service
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=dev

# Build with instrumentation
otel-go-build -o ./bin/my-service ./cmd/my-service

# Run and generate some traffic, then check your Tempo instance

The spans will appear in Tempo within seconds if your Collector pipeline is running. Compare the span coverage against what your current APM tool shows for the same service. That comparison is the most useful signal for deciding whether the coverage is sufficient for your use case.

Where to Go Next

The OTel Go compile-time instrumentation v1 release notes are worth reading in full — the project team is transparent about what's covered and what isn't. The library coverage list is in the README of the go.opentelemetry.io/auto module.

If you're in the middle of evaluating a move off Datadog or New Relic and the Go instrumentation gap is one of your open questions, we've done this migration enough times to have a clear picture of where the real complexity lives. It's usually not the instrumentation — it's the alerting rule translation and the org-level dashboarding conventions that take the most time. We're happy to talk through your specific stack. You can reach us at etalon.systems.