Skip to content

Commit 5511414

Browse files
HDDS-16048. [Docs] Improve Tracing Docs (#522)
1 parent 61a9617 commit 5511414

2 files changed

Lines changed: 71 additions & 0 deletions

File tree

docs/05-administrator-guide/03-operations/09-observability/06-distributed-tracing.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,32 @@ Tracing is turned off by default. To enable it across Ozone services, configure
1919
</property>
2020
```
2121

22+
## Application-Aware Client Tracing
23+
24+
Application-aware tracing lets Ozone participate in an existing application trace without starting its own
25+
root traces when cluster-wide tracing is disabled. It applies only when `ozone.tracing.enabled=false` and parent context is passed.
26+
27+
This is controlled by:
28+
29+
```xml
30+
<property>
31+
<name>ozone.tracing.client.application-aware</name>
32+
<value>true</value>
33+
</property>
34+
```
35+
36+
**Default Value:** `true`
37+
38+
**Behavior:**
39+
40+
- When `true`, the Ozone client can create child spans if an application trace is already active
41+
(via the application's Global OpenTelemetry instance or a W3C-propagated context).
42+
Ozone will not start a new root trace on its own.
43+
- When `false` (with `ozone.tracing.enabled=false`), client tracing is fully off.
44+
- When `ozone.tracing.enabled=true`, Ozone uses its own OpenTelemetry SDK and exports
45+
spans normally; application-aware mode does not change that behavior.
46+
- Set `ozone.tracing.endpoint` on the Ozone side to the same OTLP collector endpoint used by your application.
47+
2248
## Configuration Priorities
2349

2450
When resolving configurations for endpoints and sampling strategies, Ozone evaluates sources in the following order of priority:
@@ -96,6 +122,51 @@ export OTEL_SPAN_SAMPLING_ARG="createVolume:1.0,getBucket:0.5"
96122

97123
> **Note:** In this example, 100% of `createVolume` spans and 50% of `getBucket` spans will be collected.
98124
125+
## Instrumented Components
126+
127+
When tracing is enabled, specific services emit spans using the designated identifiers below.
128+
Trace context is propagated across service boundaries via gRPC and W3C context propagation.
129+
130+
| Service / Component | Service Name |
131+
|---------------------------| ------------ |
132+
| Ozone Manager | `OzoneManager` |
133+
| Storage Container Manager | `StorageContainerManager` |
134+
| Datanode | `HddsDatanodeService.{datanodeId}` |
135+
| S3 Gateway | `S3gateway` |
136+
| Ozone Client | `client` (when Ozone initializes tracing in the JVM) |
137+
| CLIs (Shell / FS / Freon) | `shell`, `FsShell`, `freon` |
138+
139+
> **Note:** If an application registers OpenTelemetry first, client spans are exported under that application's service name, not `client`.
140+
141+
## Dynamic Reconfiguration
142+
143+
You can update the following tracing properties at runtime on the OM, SCM, and Datanodes
144+
without restarting the processes:
145+
146+
- `ozone.tracing.enabled`
147+
- `ozone.tracing.endpoint`
148+
- `ozone.tracing.sampler`
149+
- `ozone.tracing.span.sampling`
150+
- `ozone.tracing.client.application-aware`
151+
152+
> **Note:** S3 Gateway and the Ozone client do not support dynamic reconfiguration.
153+
154+
For more details on dynamic property reload, see [Dynamic Property Reload](../dynamic-property-reload).
155+
156+
## Quick Start
157+
158+
1. Start your Ozone cluster and a Jaeger collector.
159+
2. Enable tracing and set the collector endpoint to your Jaeger OTLP receiver. See [Enabling Tracing](#enabling-tracing) and [Collector Endpoint Configuration](#collector-endpoint-configuration).
160+
3. Generate sample traces:
161+
162+
```shell
163+
ozone freon rk --numOfVolumes=1 --numOfBuckets=1 --numOfKeys=2
164+
```
165+
166+
4. Open the Jaeger UI, select a service such as `OzoneManager` or `freon`, and click **Find Traces**.
167+
168+
![Jaeger trace view showing freon command](jaeger-trace.png)
169+
99170
## References
100171

101172
- Design doc: [HDDS-13679 Distributed tracing improvement](https://github.com/apache/ozone/blob/master/hadoop-hdds/docs/content/design/distributed-tracing-OpenTelemetry.md)
390 KB
Loading

0 commit comments

Comments
 (0)