Skip to content

perf(epp): derive Scope allowed-key sets once at datalayer init - #2471

Merged
LukeAVanDrie merged 2 commits into
llm-d:mainfrom
LukeAVanDrie:epp-scope-memoize
Aug 21, 2026
Merged

perf(epp): derive Scope allowed-key sets once at datalayer init#2471
LukeAVanDrie merged 2 commits into
llm-d:mainfrom
LukeAVanDrie:epp-scope-memoize

Conversation

@LukeAVanDrie

@LukeAVanDrie LukeAVanDrie commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

datalayer.Scope rebuilt the allowedPut/allowedGet maps from Produces()/Consumes() on every invocation. Scope runs for every filter, every scorer, and every DataProducer on every request, while the declarations are fixed at plugin construction.

The sets are now derived once at startup: the runner calls datalayer.RegisterScopeSpecs(handle.GetAllPlugins()) next to ValidateAndOrderDataDependencies, after auto-created producers exist. Scope looks the spec up by the plugin's typed name, the same key ValidateAndOrderDataDependencies uses. A plugin that never went through registration is confined to nothing, with a one-time error log, so a wiring bug fails closed.

Benchmarks (benchstat, 10 interleaved runs per side, Apple M4 Pro):

                      │     main      │  this PR      vs base                │
BenchmarkScope sec/op
Scope/endpoints=10      533.4n ± 3%    318.2n ± 4%   -40.34% (p=0.000 n=10)
Scope/endpoints=100     1.733µ ± 4%    1.640µ ± 6%    -5.40% (p=0.018 n=10)

BenchmarkScope allocs/op
Scope/endpoints=10      10.000 ± 0%    7.000 ± 0%    -30.00% (p=0.000 n=10)
Scope/endpoints=100     10.000 ± 0%    7.000 ± 0%    -30.00% (p=0.000 n=10)

Which issue(s) this PR fixes:

Part of #2470

Release note (write NONE if no user-facing change):

NONE

Test plan

  • New unit tests: an unregistered plugin is confined to nothing (declared reads and writes both denied, without panicking); re-registering a typed name replaces its spec
  • Existing ./pkg/epp/datalayer/..., ./pkg/epp/scheduling/..., ./pkg/epp/requestcontrol/..., disagg profile-handler, and ./cmd/epp/... unit tests (tests that assert scoped data flow now register their plugins)
  • make presubmit stages (signed-commits-check trips on the pre-existing GitHub merge commit 843d2271, which the DCO and signed-commit CI checks exempt)
  • BenchmarkScope before/after with benchstat

@LukeAVanDrie
LukeAVanDrie requested a review from a team as a code owner August 19, 2026 23:33
@LukeAVanDrie
LukeAVanDrie requested review from ahg-g and vMaroon August 19, 2026 23:33
@github-actions github-actions Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/cleanup area/epp area/datalayer and removed kind/cleanup labels Aug 19, 2026
Comment thread pkg/epp/datalayer/endpoint_scope.go Outdated
// the process lifetime, so entries are never evicted.
var scopeSpecs sync.Map // fwkplugin.Plugin -> *scopeSpec

func scopeSpecFor(plugin fwkplugin.Plugin) *scopeSpec {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an init logic, I recommend to call this explicitly from the datalayer init code instead of doing it on the fly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The sets are now derived once at startup: the runner calls datalayer.RegisterScopeSpecs(handle.GetAllPlugins()) next to ValidateAndOrderDataDependencies, after auto-created producers exist, and Scope looks the spec up by the plugin's typed name (the same key ValidateAndOrderDataDependencies uses).

A plugin that never went through registration is confined to nothing, with a one-time error log, so a wiring bug fails closed instead of silently widening access. This also drops the lazy sync.Map and the plugin-as-map-key requirement.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated bench data in PR description too

@github-actions github-actions Bot added area/scheduling size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 21, 2026
@LukeAVanDrie LukeAVanDrie changed the title perf(epp): cache Scope allowed-key sets per plugin instance perf(epp): derive Scope allowed-key sets once at datalayer init Aug 21, 2026
Scope rebuilt the allowedPut/allowedGet maps from Produces()/Consumes()
on every invocation, though the declarations are fixed at plugin
construction and Scope runs for every filter, scorer, and DataProducer
on every request. Derive the sets once per plugin instance and share
them read-only across invocations.

BenchmarkScope: -45% time / -40% allocs at 10 endpoints, -6% / -50% at
100. Confinement behavior is unchanged.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
Scope looks up a spec registered by RegisterScopeSpecs, keyed by the plugin's typed name,
instead of caching lazily by plugin identity. The runner registers every plugin once the full
set, including auto-created producers, is known. An unregistered plugin is confined to nothing
and the miss is logged once per typed name.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
@LukeAVanDrie
LukeAVanDrie merged commit 185bb7a into llm-d:main Aug 21, 2026
40 checks passed
@LukeAVanDrie
LukeAVanDrie deleted the epp-scope-memoize branch August 21, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/datalayer area/epp area/scheduling kind/cleanup size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants