OpenTelemetry Go Compile-Time Instrumentation v1: What It Means for Your Self-Hosted Stack

The OpenTelemetry Go compile-time instrumentation library reached v1 this month. If you have spent any time trying to auto-instrument Go services the way you can with Java agents or Python's sitecustomize, you know how painful the alternatives have been. Go's static linking and lack of a runtime bytecode model made eBPF the only credible zero-code path — and eBPF has its own production constraints. Compile-time instrumentation is a different bet: modify the binary at build time, not at runtime. v1 signals the project considers the API stable enough to depend on.

This post is a technical look at what the v1 release actually delivers, how it fits into a self-hosted stack (Grafana + Tempo + Mimir + Loki on AWS), and where the tradeoffs land. We have been testing this approach with a handful of Go-heavy services over the past few months. The results are interesting but not universally positive.

What Compile-Time Instrumentation Actually Does

The Go OTel compile-time library uses a toolexec-based approach. You pass -toolexec to the Go compiler, and the instrumentation tool intercepts each package compilation, injects OTel spans and metrics at the AST level, then hands the modified source to the real compiler. The result is a normal Go binary — no sidecar, no LD_PRELOAD, no eBPF probe — that emits OTLP traces and metrics.

The practical consequence: you can instrument a third-party library you do not own without forking it, as long as you control the build.

Here is the minimal build invocation:

go build -toolexec="otel-go-instrumentation" -o myservice ./cmd/myservice

You configure which packages to instrument via a YAML manifest:

# otel-instrumentation.yaml
instrument:
  - package: net/http
    spans:
      - function: (*Transport).RoundTrip
        kind: client
      - function: (*ServeMux).ServeHTTP
        kind: server
  - package: database/sql
    spans:
      - function: (*DB).QueryContext
        kind: client
        attributes:
          - name: db.statement
            from: arg1  # the query string
exporter:
  otlp:
    endpoint: "http://otel-collector.monitoring.svc:4317"
    protocol: grpc

The instrumentation manifest ships with the binary as an embedded resource. At startup, the binary reads it and registers the injected hooks. There is no separate config reload path — changing instrumentation means a rebuild.

Why This Matters for Self-Hosted Stacks

If you run Grafana Cloud, you have a managed OTel pipeline and the instrumentation story is largely handled for you. If you run your own Grafana + Tempo + Mimir + Loki on EKS, you own the entire chain from SDK to storage. The compile-time approach changes a few things in that chain.

Collector Topology Stays the Same

The binary emits standard OTLP. Your existing OpenTelemetry Collector deployment does not change. We run a Collector DaemonSet on EKS with a pipeline that fans out to Tempo (traces), Mimir (metrics via OTLP ingest), and Loki (structured logs via the logtransform processor). Compile-time instrumented services slot into that pipeline identically to manually instrumented ones.

# otel-collector-config excerpt (Collector v0.105+)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  resource:
    attributes:
      - key: deployment.environment
        from_attribute: k8s.namespace.name
        action: insert

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.monitoring.svc:4317
    tls:
      insecure: true
  prometheusremotewrite/mimir:
    endpoint: http://mimir-distributor.monitoring.svc:8080/api/v1/push
  loki:
    endpoint: http://loki-distributor.monitoring.svc:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [prometheusremotewrite/mimir]

Build Pipeline Impact

This is where it gets real. The -toolexec flag adds a compilation step per package. In our testing on a mid-sized Go service (roughly 80k lines, 40 internal packages, 120 dependencies), clean build time increased from 47 seconds to 68 seconds — a 45% increase. Incremental builds are less affected because unchanged packages are cached, but any change that invalidates the cache for a heavily-instrumented package (like net/http) re-runs the injection.

If you use a build cache (and you should — GOCACHE on a shared EFS volume or an S3-backed Bazel remote cache), the steady-state overhead drops to near zero for unchanged packages. The first build after a tooling update is the expensive one.

These numbers are from an m6i.2xlarge CI runner. Your numbers will vary, but the pattern holds: cold builds hurt, warm builds do not.

Binary Size

The injected code adds OTel SDK dependencies to your binary whether or not you had them before. For a service that was previously zero-dependency on OTel, expect binary size to increase by 8-14 MB (the SDK, exporters, and OTLP proto definitions). For services already importing go.opentelemetry.io/otel, the delta is smaller — the linker deduplicates.

What v1 Stabilizes (and What It Does Not)

The v1 release stabilizes the instrumentation manifest format and the hook injection API for packages that have explicit support. The project ships built-in support for:

• net/http (client and server)
• database/sql
• google.golang.org/grpc
• github.com/gorilla/mux
• github.com/gin-gonic/gin
• github.com/go-redis/redis/v9

For packages not on that list, you can write custom injection rules, but that requires understanding the AST transformation API, which is not yet stable. The project is explicit about this: the public manifest format is v1, the extension API is not.

What this means practically: if your Go services use the standard library and one of the supported frameworks, you can depend on this today. If you have internal RPC frameworks or unusual database drivers, you are writing custom rules against an unstable API. We would not do that in production yet.

The eBPF Comparison

The main alternative for zero-source-change Go instrumentation is the OpenTelemetry eBPF auto-instrumentation project (the kernel-side one, confusingly also called opentelemetry-go-instrumentation). It uses uprobes to intercept function calls at runtime.

Here is how they compare for a self-hosted EKS deployment:

If you run Fargate, compile-time is your only zero-code option. eBPF requires a privileged DaemonSet with CAP_SYS_ADMIN or CAP_BPF, which many security teams reject. Compile-time instrumentation has no elevated runtime privileges.

If you run on EC2 node groups with Bottlerocket and your security posture allows privileged pods, eBPF is still a reasonable choice — especially for services you genuinely cannot rebuild (third-party binaries, vendor-supplied containers). Compile-time does not help you there.

Trace Correlation with Loki Logs

One thing compile-time instrumentation enables cleanly is trace-to-log correlation. Because the injected spans run inside the same process as your logger, you can propagate trace context into your structured log fields. With Loki as your log backend and Tempo as your trace backend, this gives you the full Grafana Explore correlation flow.

With go.uber.org/zap and the OTel SDK:

// This is the pattern the injected code follows for supported loggers.
// If you use zap, the compile-time injector adds this automatically
// for instrumented handlers.
func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        span := trace.SpanFromContext(r.Context())
        sc := span.SpanContext()
        logger := zap.L().With(
            zap.String("trace_id", sc.TraceID().String()),
            zap.String("span_id",  sc.SpanID().String()),
        )
        ctx := ctxzap.ToContext(r.Context(), logger)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

The compile-time injector handles this automatically for net/http handlers when you use a supported logger. The trace_id field lands in your Loki log line, and Grafana's derived fields configuration picks it up:

# Loki data source derived fields (Grafana provisioning)
- name: TraceID
  matcherRegex: '"trace_id":"(\w+)"'
  url: '${__value.raw}'
  datasourceUid: tempo-prod

This is not new — you could do it manually before. The value of compile-time injection is that you stop relying on every developer remembering to thread the context and add the log fields.

Where This Goes Wrong

Three failure modes we have hit or expect to hit:

1. Build reproducibility. The instrumentation tool version must be pinned and reproducible. If your CI pulls the latest version of otel-go-instrumentation without a lockfile, you will get non-deterministic binary output. Pin the tool version in your go.mod toolchain or as a checked-in binary. We use a Makefile target that pulls a specific digest from a private ECR mirror.

2. Panic attribution. When a panic occurs in an instrumented function, the stack trace includes injected frames. These frames reference generated file paths that do not exist in your source tree. Your existing panic aggregation tooling (Sentry, or a custom Loki alert) may produce confusing stack traces. We have not found a clean solution yet — the project's issue tracker has an open discussion on this.

3. Interface satisfaction at build time. The AST injector adds method calls to types. In rare cases involving interface embedding and unexported types, the injector generates code that does not satisfy an interface the original code expected. We hit this once with a custom http.ResponseWriter wrapper. The error is caught at compile time, not runtime, but it requires a workaround in the instrumentation manifest (an exclusion rule for that specific type).

None of these are blockers, but they are real. Factor in time for debugging build-time surprises when you roll this out.

Recommendation

For Go services on Fargate, or in environments where eBPF is blocked by security policy, compile-time instrumentation v1 is production-ready for the supported package list. The build overhead is real but manageable with caching. The trace-to-log correlation story with Tempo and Loki is clean.

For services on EC2 node groups where eBPF is already running, there is no strong reason to switch. The runtime overhead is comparable, and eBPF does not require touching your build pipeline.

For services using unsupported frameworks or internal RPC systems, wait. The extension API is not stable, and writing custom AST injection rules against an unstable interface is a maintenance liability.

The Go instrumentation story in OpenTelemetry has been the weakest part of the ecosystem for years. v1 of compile-time instrumentation is a meaningful step. It is not the end of the story — the Java agent is still more capable — but it is the first Go auto-instrumentation approach we would recommend for production without significant caveats.

Next Steps

If you want to test this against your own Go services, start with the official getting-started guide and the v1 release notes. The manifest format is documented and stable. Budget an afternoon to understand the build pipeline changes before committing to a rollout.

If you are running a self-hosted Grafana + Tempo + Mimir + Loki stack on AWS and want to understand whether compile-time instrumentation fits your current collector topology, we are happy to look at your setup. At Etalon, we have been running these migrations for a while, and the instrumentation layer is usually where the surprises live. Book a discovery call if you want a second opinion.