Most observability migration guides assume you control the codebase. You don't always. Here's how we instrument legacy Java monoliths — some of them 15 years old, running on JDK 8, with no test coverage and a deployment freeze — using the OpenTelemetry Java agent and a handful of environment variables, without a single line of application code changing.
This post covers the full path: agent attachment, collector configuration, shipping traces to Tempo and metrics to Mimir, and the specific failure modes we've hit in production. OpenTelemetry graduated as a CNCF project this month, which means the "is it stable enough" conversation is over. The question now is execution.
The Actual Problem With Legacy Instrumentation
When a company is spending $800K/year on Datadog and wants to move off it, the greenfield services are easy. You swap the SDK, update the exporter endpoint, done. The hard part is always the services nobody wants to touch: the billing engine written in 2009, the order management system that three engineers understand, the batch processor that runs once a night and, if it fails, someone gets paged at 3am.
These services share a few properties:
- JDK 8 or JDK 11, rarely newer
- Deployed as fat JARs or WAR files on EC2, not containers
- No CI/CD pipeline that would let you safely add a dependency
- Often instrumented with a legacy APM agent (Datadog Java agent, AppDynamics, New Relic) that gets attached via
JAVA_TOOL_OPTIONS
That last point is the unlock. If there's already a -javaagent flag in the environment, you can swap it for the OpenTelemetry agent. No code change. No build change. A configuration change in your deployment system, a restart, and you have traces.
What the OpenTelemetry Java Agent Actually Does
The OTel Java agent (current stable: 2.x, targeting Java 8+) uses bytecode instrumentation via the Java Instrumentation API. At JVM startup, the agent rewrites bytecode for ~100 libraries — JDBC, Servlet, Spring, Hibernate, Apache HttpClient, gRPC, Kafka, and more — to inject trace context propagation and span creation.
This is exactly what Datadog's Java agent does. The mechanism is identical. The difference is that the OTel agent emits OTLP instead of Datadog's proprietary wire format.
For a service that already has a Datadog agent attached, the migration is:
# Before
JAVA_TOOL_OPTIONS="-javaagent:/opt/dd-java-agent.jar"
DD_AGENT_HOST=datadog-agent.internal
DD_SERVICE=order-management
DD_ENV=production
# After
JAVA_TOOL_OPTIONS="-javaagent:/opt/opentelemetry-javaagent.jar"
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.internal:4317
OTEL_SERVICE_NAME=order-management
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=platform
That is the entire application-side change. Everything else is collector and backend configuration.
The Collector Architecture We Use on AWS
We run the OpenTelemetry Collector as a sidecar on EC2 instances using a systemd unit, not as a centralized fleet-wide aggregator. For legacy EC2 workloads, the centralized model introduces a single point of failure and a network hop that complicates debugging. One collector per host is operationally simpler and failure-isolated.
The collector config for a typical legacy Java host:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 1000
memory_limiter:
check_interval: 1s
limit_mib: 256
spike_limit_mib: 64
resource:
attributes:
- key: host.name
from_attribute: host.name
action: upsert
- key: cloud.region
value: eu-west-1
action: insert
exporters:
otlp/tempo:
endpoint: https://tempo.internal.etalon.systems:4317
tls:
insecure: false
ca_file: /etc/ssl/certs/internal-ca.crt
prometheusremotewrite:
endpoint: https://mimir.internal.etalon.systems/api/v1/push
headers:
X-Scope-OrgID: legacy-prod
tls:
insecure: false
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, resource, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, resource, batch]
exporters: [prometheusremotewrite]
The memory_limiter processor is not optional. On a host running a legacy Java service that already consumes 4-6GB of heap, an unbounded collector will get OOM-killed by the kernel during a traffic spike, exactly when you need it most. 256MB limit with a 64MB spike allowance is conservative and has never caused us data loss in practice — the agent buffers in-process when the collector applies backpressure.
JDK 8 Compatibility: What Actually Breaks
The OTel Java agent 2.x supports Java 8+, but "supports" is doing some work there. We've hit three specific issues:
1. TLS handshake failures with OTLP/gRPC on JDK 8
JDK 8u261 and earlier have limited TLS 1.3 support. If your collector is behind an ALB or NLB with TLS 1.3 enforced, the agent will fail to connect silently — no exception in the application log, just missing spans. Fix: either pin TLS 1.2 on the collector endpoint, or update the JDK to 8u281+. We prefer the JDK update. If the deployment freeze prevents that, use OTLP/HTTP instead of gRPC; it has better TLS negotiation fallback behavior.
2. Classloader conflicts with application-bundled libraries
Fat JARs that bundle their own version of okhttp or netty sometimes conflict with the agent's internal shaded copies. Symptom: ClassCastException at startup, service fails to start. Fix: add -Dotel.javaagent.extensions.class.loader.matcher.excludes=com.example.internal to exclude the conflicting package from instrumentation. In the worst case we've seen, we had to disable the HTTP client instrumentation entirely for one service and rely on trace context propagation from the upstream caller.
3. Excessive span volume from JDBC batch operations
An ORM doing batch inserts of 10,000 rows will generate 10,000 spans if each statement is instrumented individually. This is technically correct behavior, but it produces 50MB of trace data per minute from a single service and buries the useful signal. Fix: configure span filtering at the collector level, or set OTEL_INSTRUMENTATION_JDBC_STATEMENT_SANITIZER_ENABLED=true combined with OTEL_TRACES_SAMPLER=parentbased_traceidratio with a ratio of 0.1 for batch jobs.
Connecting Traces to Logs in Loki
The agent injects trace_id and span_id into MDC (Mapped Diagnostic Context) for Log4j and Logback automatically. For services already writing structured JSON logs, this means trace correlation in Loki is zero-config — the fields appear in the log stream and Grafana's Loki datasource can link them to Tempo.
For services writing unstructured logs (yes, still common in 2026 on legacy systems), you need a Promtail pipeline stage to extract the trace ID:
pipeline_stages:
- regex:
expression: '.*trace_id=(?P<trace_id>[a-f0-9]{32}).*'
- labels:
trace_id:
- structured_metadata:
trace_id:
Once trace_id is in Loki's structured metadata, the Grafana Tempo datasource can do bidirectional linking: from a trace span to the correlated log lines, and from a log line to the originating trace. For legacy services where the only debugging tool was log grep, this is the single most impactful observability improvement we deliver.
Metrics: What You Get for Free
The OTel Java agent emits JVM metrics out of the box when you enable the metrics pipeline. These map directly to Prometheus metric names:
| OTel metric | What it measures | Mimir/Prometheus name |
|---|---|---|
process.runtime.jvm.memory.usage |
Heap and non-heap usage by pool | process_runtime_jvm_memory_usage_bytes |
process.runtime.jvm.gc.duration |
GC pause time by collector | process_runtime_jvm_gc_duration_seconds |
process.runtime.jvm.threads.count |
Thread count by state | process_runtime_jvm_threads_count |
process.runtime.jvm.cpu.utilization |
JVM CPU usage | process_runtime_jvm_cpu_utilization_ratio |
http.server.request.duration |
HTTP server latency histogram | http_server_request_duration_seconds |
For a legacy service that previously had no metrics at all — just logs and Datadog APM traces — getting JVM GC metrics and HTTP latency histograms without touching code is significant. We've found GC pause metrics alone have identified two production incidents in the first month after migration: one caused by a heap sizing misconfiguration that had been silently degrading throughput for years.
When This Approach Is Wrong
Bytecode instrumentation is not free. The agent adds 50-150ms to JVM startup time, which matters for Lambda functions and short-lived batch jobs but is irrelevant for long-running services. More importantly:
If the service is genuinely unmaintainable, adding observability doesn't help. We've been asked to instrument services where the team's actual goal was to get enough data to justify a rewrite. That's a legitimate use case, but be honest about it — you're not improving the service, you're building the business case to replace it.
If the service runs on JDK 6 or earlier, the agent won't attach. JDK 6 is end-of-life and genuinely unsupported. The only option is log-based observability via Loki and whatever metrics the JVM exposes via JMX, scraped with jmx_exporter.
If you're in a regulated environment with strict change management, a -javaagent flag addition counts as a configuration change and may require a change request even if no code changes. We've had migrations delayed by six weeks because of this. Factor it into your timeline.
If the service is already heavily sampled in Datadog at 1%, moving to 100% trace collection with OTel will increase your Tempo storage costs. Either implement head-based sampling in the collector from day one, or accept the cost delta during the transition period and tune sampling afterward. Don't be surprised by it.
The Security Angle
OTel's CNCF graduation coincided with increased focus on supply chain security for the agent itself. For production deployments, we pin the agent JAR version and verify the SHA-256 checksum in our deployment scripts:
OTEL_AGENT_VERSION="2.5.0"
OTEL_AGENT_SHA256="e3b0c44298fc1c149afb..." # full hash from GitHub release
wget -q "https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v${OTEL_AGENT_VERSION}/opentelemetry-javaagent.jar" \
-O /opt/opentelemetry-javaagent.jar
echo "${OTEL_AGENT_SHA256} /opt/opentelemetry-javaagent.jar" | sha256sum --check
This is especially relevant for legacy services in regulated industries where the agent is being attached to a service that processes PII or financial data. The agent has read access to all JVM memory. Treat it with the same scrutiny you'd apply to any third-party library in the application itself.
What the Migration Timeline Looks Like
Based on migrations we've run for clients with 20-60 legacy Java services:
- Week 1-2: Agent deployment and basic trace/metric collection. Validate data in Tempo and Mimir, build initial dashboards in Grafana.
- Week 3-4: Log correlation via Loki. Tune sampling rates. Identify the JDBC and batch operation span volume issues and suppress them.
- Week 5-6: Decommission Datadog agent from legacy services. Validate alert parity — every Datadog monitor that existed should have a Grafana alert equivalent.
- Week 7-8: Cost reconciliation. Datadog billing for legacy services drops. Tempo and Mimir storage costs are measurable and usually 60-75% lower than the equivalent Datadog APM spend for the same services.
The critical path is almost never technical. It's getting the change management approval to touch the deployment configuration for services that nobody wants to restart.
Where to Go From Here
The OpenTelemetry Java agent documentation covers the full list of instrumented libraries and all configuration options. The instrumentation list is worth reading before you start — if your service uses a library that isn't on the list, you'll need a manual span around it.
For the collector configuration, the OpenTelemetry Collector contrib processors are where most of the production tuning happens. The transform processor in particular is useful for normalizing attribute names across legacy services that were instrumented inconsistently over the years.
If you're working through a Datadog-to-Grafana migration and the legacy Java instrumentation is the piece that's blocking you, we've done this enough times to have a repeatable playbook. The contact details are at etalon.systems. We don't do discovery calls that are really sales calls — if you come with a specific problem, we'll give you a specific answer.