Habr Breaks Down APM Auto-Instrumentation: Where a 4-Second Request Loses Its Time

The request takes 4 seconds. In a monolith, a developer would turn on a profiler and find the heavy method in an evening. In a system where a request passes through an API Gateway, several microservices, a message queue, a database and an external API, a profiler on a single machine shows only a piece of the picture. Habr has published a walkthrough of how APM answers the question "where did the time go" using distributed tracing and automatic instrumentation.

Trace and span: how the picture is assembled

In distributed tracing, each request is represented as a trace, meaning its full route through all components. A trace is divided into spans: each individual call on the frontend, backend or message bus gets its own time segment. A span has an identifier, a reference to its parent, a start time and a duration. These links form a call tree.

For spans from different services to land in the same tree, the context has to travel with the request. Today this is usually the traceparent header from the W3C Trace Context standard. It carries the trace ID (16 bytes), the current span ID (8 bytes) and flags, such as a sampling indicator. A service receives the header, creates a child span and passes the context on.

How auto-instrumentation works

The manual approach looks like this: a developer wraps calls in tracing code in every service. In teams with dozens of services, this quickly turns into technical debt. Auto-instrumentation removes that work: an agent hooks into the runtime and adds measurements around known libraries on its own.

The mechanisms depend on the platform:

• Java: the agent uses -javaagent and java.lang.instrument to modify bytecode at class load time, often built on ASM or Byte Buddy.
• .NET: the CLR Profiling API is used.
• Python and Node.js: functions in popular libraries are replaced (monkey-patching, module load hooks).
• Go and other compiled languages: eBPF or manual wrappers are more common, because the runtime is less pliable.

The agent knows the typical points: HTTP servers and clients, database drivers, queue clients. It does not need to understand business logic to time an incoming request, a SQL query or a message publish. These measurements become spans, which go to a collector and then to the APM storage.

Where this approach hits its limit

Auto-instrumentation sees only what has been described in advance. If time goes into a custom algorithm inside a method, a lock on a shared resource or a thread pool, the standard hooks show just one long span with no explanation. Additional spans around critical sections and thread profiling help here.

Overhead is a separate topic. The agent adds work on every call, so tracing is almost always combined with sampling. Head-based sampling decides a trace's fate at the start of the request and is cheap. Tail-based sampling decides after the request completes, so it keeps slow and failed requests, but it requires buffering in the collector.

In practice: where to start

If a system has more than three or four services and incidents are investigated manually through logs, it is worth starting with a minimal set of steps:

• Enable auto-instrumentation on one critical path, such as checkout, rather than across the whole infrastructure at once.
• Verify end-to-end propagation of traceparent through the gateway, queues and asynchronous handlers: traces break there most often.
• Manually add spans around the sections where the auto agent shows a "black box".
• Configure sampling so that slow requests (for example, above p95) are always kept.
• Cross-check the result against metrics and logs using a shared trace ID, so that moving between signals takes seconds.

This approach turns "the system is slow" into specifics: 3,1 seconds went to the external API, 0,6 to the database query, and the rest to the queue. Optimization starts there, not with guesses.