Pyroscope tags don't show up

[NiklasRosenstein]

Nyl 0.10.2 introduced support for NYL_PYROSCOPE_URL which can be used to have it send continuous profiling data to a Pyroscope server.

In some places in the code, we use the pyroscope.tag_wrapper() context manager to annotate stack traces taken from the thread inside the manager with additional tags.

Unfortunately it seems these tags don't make it into Pyroscope.

[mahesh-sadupalli]

I'd like to take a look at this one. I've gone through the relevant code and have a starting hypothesis.

Current usage:

• init_pyroscope() in src/nyl/tools/pyroscope.py calls configure(tags={...}) — the global tags (version, query-string params) almost certainly do arrive in Pyroscope, since they're baked into the agent config at startup.
• The missing tags appear to be the scoped ones from tag_wrapper({"entrypoint": "nyl"}) in src/nyl/commands/__init__.py.

Likely root causes I want to investigate, in order:

1. Sample-vs-tag timing. pyroscope-io runs a background sampler thread that periodically reads stack traces. If the CLI command executes faster than one sampling window (default 10 s), tag_wrapper-scoped samples may simply never be flushed before the process exits. The nyl template happy path runs in well under a second.
2. Thread-local tag scope. tag_wrapper annotates the current thread's tag context. If pyroscope-io's sampler thread doesn't honor the caller's tag context (which has been a known issue in older 0.8.x releases), tags set via the context manager get dropped even though the global ones survive.
3. Final-flush on exit. The Pyroscope agent needs to flush profile chunks before the process exits. If there's no explicit shutdown()/stop() call, tagged data accumulated in the last partial window can be lost. I didn't see a shutdown hook in init_pyroscope().
4. Pyroscope server side. Worth confirming whether the tag arrives at the server at all (visible in /labels API on the Pyroscope side) vs. arriving but not being queryable in the UI.

Plan to reproduce:

• Run Pyroscope locally via Docker, point NYL_PYROSCOPE_URL at it, run a nyl template against a small manifest, then:
  • Hit GET /pyroscope/label-values?label=entrypoint to see whether the tag reaches the server at all.
  • Try a long-running command (e.g. an apply that takes > 30 s) to rule out (1).
  • Add an explicit pyroscope.shutdown() (or equivalent flush) on exit to test (3).

Question for you: when you reported this, did you ever see the entrypoint tag appear for long-running nyl invocations, or never at all? That would let me prioritise (1)/(3) vs (2).

If it turns out to be an upstream pyroscope-io issue, I'll capture that with a minimal repro and either link a known issue or open one upstream rather than papering over it in nyl.