XRT launcher latency

[ypapadop-amd]

Summary

On npu2 (Strix Halo, AIE2P), the HSA/ROCR launch runtime (NPUDriver("hsa"))
dispatches a Triton kernel ~145–170× faster per launch than the default XRT
runtime (NPUDriver("xrt")). This is not an HSA-vs-XRT API gap, it's a
launcher implementation difference: the XRT launcher rebuilds all device
state on every dispatch, while the HSA launcher caches it in a process-global
runtime.

Measurements

Steady-state per-dispatch wall-clock (20 warmup iters, then timed; identical
bf16 matmul kernel; host wall-clock incl. host↔device copies):

Size	Runtime	min	median	mean
256³	XRT	69.4 ms	95.0 ms	93.1 ms
256³	HSA	0.43 ms	0.56 ms	1.47 ms*
512³	XRT	82.2 ms	95.8 ms	93.9 ms
512³	HSA	0.63 ms	0.66 ms	0.69 ms

* 256³ HSA mean skewed by outliers; min/median are representative.

Note XRT is ~95 ms regardless of matrix size (256³ ≈ 512³) — a hallmark of
fixed per-call setup dominating, not compute. HSA scales with problem size
(0.43 → 0.63 ms as 256→512), i.e. fixed overhead is amortized and the cost is
copy + compute.

Root cause

XRT launcher (_generate_launcher / _generate_elf_launcher in
driver.py) does the full device bring-up inside every _launch, all as
locals torn down on return:

• xrt::device(0)
• load the xclbin/elf from disk + register_xclbin / build xrt::hw_context
• resolve the kernel
• allocate BOs, map, memcpy in, sync(TO_DEVICE)
• run, sync(FROM_DEVICE), memcpy out

Nothing is cached between launches, so the ~95 ms is dominated by artifact
load + hw_context creation + BO allocation — repeated every call.

HSA launcher (hsa_launcher.py + include/HsaRuntime/HsaRuntime.cpp) keeps
a process-global HsaRuntime singleton: one hsa_init, one queue, one
completion signal, a fixed-slot kernarg pool, a pooled/reused vmem I/O buffer
free-list, and cached PDI/insts. A dispatch is just: memcpy-in → enqueue AIE
packet → wait → memcpy-out. No device/context/artifact re-setup.

XRT already has the "shared runtime" property at the library level
(libxrt_coreutil is loaded once per process), but the launcher doesn't
exploit it — it rebuilds per call. It's stateless by construction, which is also
why it never accumulates device resources (unrelated aside: that statelessness
is why XRT doesn't hit the HSA QUEUES_MAX == 1 multi-signature limit).

Mitigations

Ranked, highest-value first. These are XRT-side unless noted.

1. Cache device/context/kernel/BOs across dispatches (the big one). Mirror
   what HSA's HsaRuntime does: open xrt::device, load the xclbin/ELF, build
   the hw_context, and resolve the kernel once (per signature, or a shared
   runtime), then reuse across launches. This removes essentially all of the
   ~95 ms fixed cost. This is the same pattern as the HSA shared-runtime
   library.
2. Pool the BOs instead of allocating/mapping/freeing them every dispatch;
   reuse by size (HSA already does this for vmem buffers).
3. Avoid re-register_xclbin / hw_context rebuild per call once Remove token requirement when checking out the repo #1 caches
   the context.
4. (HSA-side, further speedup, not XRT) direction-aware copy: the HSA path
   currently copies all tensor buffers back to host (it can't tell inputs from
   outputs — Triton doesn't mark them), so read-only inputs are copied both
   ways. A per-tensor in/out hint from the launcher would skip copying inputs
   back. Tracked in docs/hsa-zero-copy-notes.md (also covers a true zero-copy
   host-pointer path via hsa_amd_memory_lock).

Caveats

• Host wall-clock; includes host↔device memcpy for both runtimes. On-device
  compute time was not isolated.
• The XRT number is implementation-bound (stateless launcher), not a
  raw-runtime-API limit. Mitigation Remove token requirement when checking out the repo #1 is expected to close most of the gap.
• Correctness verified against torch each run for both runtimes.

Environment

npu2 / Strix Halo (AIE2P), Linux; transform_aie2p.mlir tiling; bf16 matmul.