Design: ccache accuracy for release builds (fingerprint coverage vs remote cache)

[subodh-dubey-amd]

Moving the design conversation here (out of issue #7248, which stays as the general problem statement) so we can weigh approaches in the open before landing anything.

Problem

Release builds read the shared ccache remote cache, so a cached object's correctness depends on the compiler_check fingerprint capturing everything that affects output. The POSIX fingerprint (build_tools/posix_ccache_compiler_check.py) hashes the compiler binary and its ldd shared libraries, but not the device bitcode libraries (ROCm-Device-Libs *.bc) or lld. A change confined to those (clang unchanged) would not invalidate the cache, so a stale object could be reused - worst for builds whose artifacts ship (prereleases promoted to stable).

Evidence (fingerprint works, with a gap)

Recent nightlies on the release cache:

Stage (nightly)	Aug 6	Aug 7	Aug 10
compiler-runtime (LLVM) hits	95.6%	93.5%	92.3%
math-libs (gfx94X) hits	31%	27%	18%

LLVM caches well (stable host compiler); math-libs is mostly misses because each night's rebuilt clang changes the fingerprint (invalidation working). Residual concern = the uncovered inputs.

Clarified requirement (from review)

Release (prerelease) builds should have zero remote cache hits, but keep the local cache (some builds, notably Windows, need at least a local cache for performance and runner stability).

Options

Option	Remote reads	Local cache	Shipped accuracy	Cost
A. status quo	yes	yes	fingerprint-dependent	low
B. add device-libs + lld to fingerprint	yes	yes	better, still heuristic	one cold rebuild + namespace bump
C. Local-only (disable remote)	no	yes	provably fresh (no remote hit)	local-cold across fresh runners
D. recache	no	no (forced miss)	provably fresh	full recompile; loses local hits
E. disable ccache	no	no	provably fresh	full recompile + build-config drift

Local-only meets the requirement best: zero remote hits, local cache retained. recache and full-disable were considered but both kill local hits.

Prototype

Local-only for prereleases is prototyped in PR #7249 (draft, not a merge request): a --no-remote-cache flag in setup_ccache.py gated on release_type == 'prerelease'.

Open questions

1. Is local-only the right call, or harden the fingerprint (B) instead / as well?
2. Scope: prerelease only (nightly keeps the remote warm cache), or nightly too?
3. If we also do B, what else should feed the fingerprint?

[subodh-dubey-amd]

@suchitravemuri1

I have opened a discussion here.

[subodh-dubey-amd]

Based on today's standup discussion with @ScottTodd , the recommendation is to continue using a local cache, as it remains important for build performance and reliability. Scott noted that we've seen Windows runners fail midway through jobs when the self-hosted runner loses connection to the server. Additionally, removing the local cache could place excessive load on the CPU.

cc: @arjun-raj-kuppala @amd-aakash @suchitravemuri1