[FEATURE] Explore optional Opik integration for agent tracing and evaluation

[muddlebee]

Problem statement

OpenSRE has semantic instrumentation for investigations, LLM invocations, tool execution, pipeline stages, routes, and session boundaries, but those spans are currently primarily JSONL diagnostic sidecars for interactive sessions. There is no optional LLM-focused observability backend where operators can inspect agent trace trees, compare investigation quality over time, or turn production failures into repeatable evaluation datasets.

This makes it harder to answer questions such as:

• Which LLM or tool step made an investigation slow or unsuccessful?
• Did a model or prompt change improve evidence quality and root-cause accuracy?
• Can a failed production investigation become a regression case?
• Can customers keep this telemetry inside their own infrastructure?

Opik is an Apache-2.0 LLM observability and evaluation platform with agent tracing, datasets, experiments, online evaluation, OpenTelemetry ingestion, and self-hosting. It may complement OpenSRE's existing runtime telemetry without replacing Sentry, Grafana, or product analytics.

Proposed solution

Run a bounded spike to evaluate an optional, disabled-by-default Opik integration with two separately scoped capabilities:

1. Export OpenSRE agent traces to Opik

   • Prefer OTLP/HTTP through OpenSRE's existing OpenTelemetry dependencies rather than adding Opik decorators throughout business logic.
   • Preserve the existing JSONL session trace path; Opik should be an additional destination.
   • Model a trace as an investigation or conversational turn with nested LLM, tool, stage, route, and component spans.
   • Include safe metadata such as model/provider, duration, token counts, outcome, tool name, surface, and investigation/session correlation IDs.
   • Batch exports, flush on shutdown, and fail open so an unavailable telemetry backend never breaks an investigation.

2. Evaluate OpenSRE investigations with Opik

   • Explore converting curated or failed investigations into versioned datasets.
   • Compare prompt/model versions using OpenSRE-specific metrics such as evidence grounding, root-cause correctness, citation/tool-use quality, and actionability.
   • Keep evaluation out of the live request path during the initial spike.

A later, independent feature could expose Opik as an investigation source so OpenSRE can diagnose other LLM applications from their Opik traces. That should not be coupled to the exporter spike.

Safety and privacy requirements

• Honor OPENSRE_NO_TELEMETRY and any global do-not-track behavior.
• Do not export prompts, responses, incident payloads, logs, or tool inputs/outputs by default.
• Any content capture must be explicit opt-in, redacted, truncated, documented, and tested.
• Support a self-hosted endpoint; do not assume Opik Cloud.
• Keep credentials and shared environment-variable names under config/constants/.
• Do not log secrets or expose exporter failures to external chat surfaces.

Technical questions for the spike

• Should the implementation be an Opik-specific exporter or a generic LLM OTLP exporter with an Opik configuration preset?
• How should trace context and parent/child IDs be represented in platform/observability/trace/?
• How should context propagate into parallel ThreadPoolExecutor tool calls and concurrent gateway turns?
• How should multiple destinations coexist without changing JSONL behavior?
• Which OpenTelemetry GenAI semantic attributes are stable enough to adopt?
• Can token/cache usage already present in LLM response types be attached without duplicating analytics hooks?
• What sampling and retention defaults are appropriate for local, gateway, and hosted runtimes?

Definition of done for exploration

• Produce a short design note comparing native Opik SDK vs OTLP/HTTP.
• Demonstrate one local or self-hosted Opik trace with a real parent/child tree: turn/investigation → LLM invocation → tool execution.
• Verify parallel tool calls and concurrent turns remain isolated.
• Verify JSONL tracing continues to work unchanged when Opik export is enabled.
• Verify the disabled path has negligible overhead and no network activity.
• Verify exporter/network failure does not affect investigation success.
• Add privacy tests proving raw content and secrets are excluded by default.
• Document configuration, self-hosting, opt-in content behavior, and shutdown/flush behavior.
• Decide whether to proceed with production integration, revise the generic tracing layer first, or stop after the spike.

References

• Opik repository: https://github.com/comet-ml/opik
• OpenTelemetry ingestion: https://www.comet.com/docs/opik/integrations/opentelemetry
• Evaluation overview: https://www.comet.com/docs/opik/evaluation/overview
• Self-hosting overview: https://www.comet.com/docs/opik/self-host/overview/

[tmohanvamsi]

I’m interested in contributing to this spike, particularly the OTLP/HTTP export and self-hosted tracing path. Is this ready for a contributor, and would you prefer the initial scope to be the design note plus a minimal trace prototype?