tracing: configure SDK explicitly instead of mutating process env - #2298
tracing: configure SDK explicitly instead of mutating process env#2298satyamg1620 wants to merge 2 commits into
Conversation
* rename epp standalone to standalone * fixed typo * fixed name * fixed typo
|
|
||
| _, ok = os.LookupEnv("OTEL_EXPORTER_OTLP_ENDPOINT") | ||
| if !ok { | ||
| os.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317") |
There was a problem hiding this comment.
How is this default value applied now?
There was a problem hiding this comment.
Thanks @ahg-g for the review.
The SDK supplies the endpoint: NewGRPCConfig seeds localhost:4317 before it applies env config or explicit options (options.go#L115-L131). But it negotiates TLS for that default, and #2312 removed the hardcoded WithInsecure — so the http:// scheme in the old os.Setenv value was what selected plaintext. Dropping it outright broke the local collector path: tls: first record does not look like a TLS handshake, 0 spans delivered.
Pushed a fix that pins the endpoint and the insecure transport, but only when the environment sets none of the endpoint, insecure or certificate variables. As soon as one is set the options are dropped, so an operator's OTEL_EXPORTER_OTLP_ENDPOINT and its scheme still decide where spans go and how the connection is secured.
30e67a5 to
f545f72
Compare
Signed-off-by: satyamg1620 <Satyam.Gupta.3@ibm.com>
Signed-off-by: satyamg1620 <Satyam.Gupta.3@ibm.com>
83854fb to
66cf37e
Compare
What type of PR is this?
/kind bug
What this PR does / why we need it:
InitTracingapplied its defaults by callingos.SetenvforOTEL_SERVICE_NAMEandOTEL_EXPORTER_OTLP_ENDPOINT, relying on the SDK to read them back. Mutating the process environment affects any other code in the process that reads those variables, is not reentrant, and leaves tests interfering with each other —InitTracinghad no test coverage at all as a result.The resource is now built directly via
resource.New. Detectors are applied in order and later ones win, so placingWithFromEnvlast keepsOTEL_SERVICE_NAMEandOTEL_RESOURCE_ATTRIBUTESoverriding the built-in service name.This also fixes trace attribution. The previous resource was built with
resource.NewWithAttributes, which never consults the environment, so neitherservice.namenor the chart'sOTEL_RESOURCE_ATTRIBUTES(k8s.namespace.name,k8s.node.name,k8s.pod.name) ever reached a span — theos.Setenvcalls had no effect on the exported resource. Spans now carry them.Three implementation notes:
The plaintext loopback default is pinned, conditionally. The SDK defaults the gRPC endpoint to
localhost:4317but negotiates TLS for it, and tracing: honor OTLP TLS and auth environment variables #2312 removed the hardcodedWithInsecure. Simply dropping the old default therefore pointed the local development path at TLS against a plaintext collector.localCollectorOptionssupplies the endpoint and the insecure transport only when the environment sets none of the endpoint, insecure or certificate variables; as soon as one is set the options are dropped, so an operator's configuration is never overridden.A partial resource no longer aborts startup.
resource.Newreports a malformedOTEL_RESOURCE_ATTRIBUTESentry alongside a resource holding everything it could parse. Returning that error made a single stray comma in a chart value failInitTracing, which the EPP runner turns into a process exit. The error is logged and the degraded resource is used.resource.Default()is not merged in. It carries semconv v1.41.0 while this package uses v1.37.0, and merging conflicting schema URLs drops the schema URL from the result. Aligning the semconv version is a separate change; the reason is recorded in a comment so it is not re-added by accident.Which issue(s) this PR fixes:
Fixes #2297
Part of #1632.
Test plan:
service.namedefaults to the caller-supplied name when the environment is emptyOTEL_SERVICE_NAMEandOTEL_RESOURCE_ATTRIBUTESoverride that default, withOTEL_SERVICE_NAMEwinning between the twoInitTracingleaves everyOTEL_*variable untouched in the environmentOTEL_RESOURCE_ATTRIBUTESentry keeps the parseable attributes and lets initialization succeedOTEL_SERVICE_NAME,OTEL_RESOURCE_ATTRIBUTESandOTEL_EXPORTER_OTLP_ENDPOINTalready set in the environmentservice.name=llm-d-eppwith noOTEL_SERVICE_NAMEset, plusk8s.namespace.name,k8s.node.nameandk8s.pod.name, at schema URL1.37.0Release note (write
NONEif no user-facing change):