Distributed Traces
Most important commands to remember
openssl rand -hex— generate fresh test trace identifiers.curl -H— send a request carrying explicit trace context.
Commands and flags
| Command or syntax | Meaning |
|---|---|
openssl rand -hex 16 / -hex 8 |
Generate sixteen / eight random bytes represented as hexadecimal. |
trace_id=$(…) / parent_id=$(…) |
Save the generated values in shell variables. |
printf 'trace_id=%s\n' "$trace_id" |
Print the ID to search in the tracing backend. |
--silent --show-error --max-time 5 |
Hide progress, show failures, and limit curl to five seconds. |
-i -H 'HEADER' |
Include response headers and send the specified request header. |
00-TRACE-PARENT-01 |
W3C version, 32-hex trace ID, 16-hex parent ID, and sampled flag. |
This creates a synthetic caller context; it does not create or export a caller span. IDs are correlation data, not authentication credentials.
The concepts that matter
1. A trace connects work belonging to an operation
A distributed trace connects timed operations involved in a request or workflow. Each operation is a span, with identity, timing, attributes, and relationships to other spans.
A trace can show a frontend calling an API that then queries a database. This explains where the request spent time more directly than unrelated logs from each component. Asynchronous work can use links when a simple parent-child tree is insufficient.
2. Context propagation keeps the relationship intact
A service receives context, creates its own span, and propagates suitable context when making downstream calls. W3C traceparent standardizes core identifiers for HTTP propagation.
If one hop drops or replaces context, the backend may show disconnected traces. Sending a header alone creates no instrumentation: the receiving application must extract it and record/export spans. Context also crosses trust boundaries, so receivers may apply validation and sampling policy.
3. Span duration is elapsed time, not CPU consumption
A span can include waiting for a pool slot, network I/O, locks, or another service. A long span therefore does not necessarily mean expensive computation in the component that owns it.
Nested durations also overlap. Adding every span duration can exceed total request latency. Inspect the critical sequence and concurrent branches instead of summing all rows as if they were independent serial work.
4. The visible trace can be incomplete
Sampling controls which operations are recorded or retained. Export failures, queue drops, missing instrumentation, and clock differences can also affect the final view.
A sampled flag expresses tracing intent, not a promise that every service or backend will retain the trace. Missing spans are not proof that no downstream call happened. Compare tracing policy and pipeline health before drawing a complete execution story from a partial trace.
One small example
Optional: use the stated instrumented test service and its existing backend. Run the block, then search the tracing backend for the printed trace ID after allowing its normal ingestion delay.
trace_id=$(openssl rand -hex 16)
parent_id=$(openssl rand -hex 8)
printf 'trace_id=%s\n' "$trace_id"
curl --silent --show-error --max-time 5 -i -H "traceparent: 00-$trace_id-$parent_id-01" http://localhost:8080/If the service honors the incoming context, inspect a server span with that trace ID and the supplied parent ID. The synthetic parent has no exported span, so a missing caller row is expected. Downstream spans appear only where instrumentation and propagation exist.
The HTTP result and the trace search answer different questions. A successful response with no visible trace can reflect sampling or export configuration. This request proves neither complete coverage nor accurate cross-host clock ordering. No collector or application configuration is changed.
Keep this idea: A trace links timed work through propagated context; its value depends on instrumentation, sampling, and what the pipeline actually retained.