perf: Enable shared closure DP for npm and yarn1 lockfiles - #13635
Merged
Conversation
The shared transitive-closure DP computes each workspace closure once over the global package graph, but only pnpm implemented transitive_edge_resolver, so npm and yarn1 always fell back to the legacy walk that re-resolves and re-visits every edge of the shared dependency graph once per workspace. Both formats can prove edge uniformity cheaply: - npm: transitive edges come from all_dependencies, which only emits keys present in the packages map, and resolve_package matches such a key directly without consulting the workspace. The resolver mirrors that key-hit branch and conservatively reports any non-key name as workspace-sensitive (unreachable via all_dependencies output). - yarn1: resolve_package ignores its workspace argument entirely, so every edge is globally uniform by construction. New differential tests assert the DP result matches the legacy walk, including an npm case with a workspace-nested package version and a deeply nested transitive dependency. On a synthetic lockfile with 50 workspaces sharing a 4,500-package graph, all_transitive_closures drops from 0.50s to 0.09s (5.7x) for npm and from 0.44s to 0.08s (5.6x) for yarn1; the gap grows with workspace count since the legacy walk is O(workspaces x closure size). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
anthonyshew
added a commit
that referenced
this pull request
Aug 3, 2026
…ures (#13641) ### Description > **Base of a 3-PR stack**: #13641 (this) → #13642 (parallel fingerprint) → #13643 (parallel identity build). This PR targets `main`; review in order. Together they cut package-graph construction ~264 ms → ~178 ms (~33%) on the profiled repo. Profiling package-graph construction on a large real-world pnpm monorepo (~1,200 workspaces) showed `resolve_dependencies` materializing every workspace's external closure independently: one `ExternalPackageIdentity` per closure member per workspace (two string clones plus a `human_name` lockfile lookup each — ~390k constructions total), then one fingerprint hash per workspace over the full `(key, version)` sequence. Only ~two-thirds of those closures are actually distinct — the rest is duplicated work. Changes: - `PackageResolution` now stores its identity list as `Arc<[ExternalPackageIdentity]>`. The list is sorted/deduped at construction (`new`, the only constructor) and immutable afterward. - `resolve_dependencies` shares one materialized list across workspaces whose closures are identical, detected by the closure's `Arc` pointer sequence. Closure members are interned `Arc`s whenever the shared closure DP produced them (pnpm today; npm/yarn1 with #13635), so identical pointer sequences ⇔ identical closures. Legacy-walk formats produce distinct pointers per workspace and simply keep building independently — behavior unchanged, no penalty. - `ExternalResolutionGeneration::build` fingerprints each distinct list once, memoized by slice address — at most one map probe per *package* (not per member), so unshared lists pay nothing measurable beyond the hash they already required. Its per-package re-sort is dropped since `new` already guarantees the invariant. Public surface is unchanged: `identities()` still returns `&[ExternalPackageIdentity]`, `new()` has the same signature, and cargo/uv resolution producers are untouched. Cloning a resolution/generation is now a refcount bump instead of a deep identity-list copy. **Measured** on the profiled repo (release, 8 warm samples per side): package graph construction ~264 ms → ~249 ms median (**~6%**) from this PR alone, with a ~35% cut in identity-list allocations. The two PRs stacked on top take it to ~178 ms. For transparency: two cheaper alternatives were tried first — per-member identity memoization and content-keyed fingerprint dedup — and both measured neutral-to-worse (per-member map probes cost what they saved), which is why this lands as the structural pointer-keyed sharing instead. ### Testing Instructions - `cargo test -p turborepo-repository` — all 419 tests pass (identity ordering/dedup through `new`, generation building, cross-domain validation). - `cargo check -p turborepo-lib` compiles unchanged against the new field representation. - `cargo clippy -p turborepo-repository --all-targets` is clean. - Fingerprint semantics are unchanged: the hash input is the same sorted, deduped `(key, version)` sequence as before; sharing only avoids recomputing it for byte-identical sequences. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU Co-authored-by: Claude <noreply@anthropic.com>
tknickman
approved these changes
Aug 3, 2026
anthonyshew
pushed a commit
that referenced
this pull request
Aug 5, 2026
## Release v2.10.9-canary.1 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/31044275876) ### Changes - refactor: Delete legacy external-resolution PackageInfo state (#13526) (`fbe88d3`) - chore: Release Turborepo 2.10.8-canary.4 (#13557) (`887d9b1`) - refactor: Remove external declaration compatibility paths (#13527) (`3c8448d`) - refactor: Produce immutable native task and command knowledge (#13528) (`18d0bc3`) - refactor: Migrate native task registration and suggestions (#13529) (`7b5ee63`) - refactor: Migrate turbo-json native task synthesis (#13530) (`2d004f0`) - refactor: Migrate persistent and recursive task validation (#13531) (`dd7519b`) - refactor: Migrate native task definition precedence (#13532) (`8792ce9`) - refactor: Migrate engine native command planning (#13533) (`67e4e9e`) - refactor: Migrate executor native command resolution (#13534) (`1b8accc`) - refactor: Migrate native task query, devtools, and LSP views (#13535) (`2753b76`) - refactor: Migrate command summaries and delete legacy task paths (#13536) (`adbad50`) - refactor: Produce immutable task-contract knowledge (#13537) (`97bff31`) - refactor: Migrate engine task-contract composition (#13538) (`fab50c5`) - refactor: Migrate hashing engines to task contracts (#13539) (`76206f8`) - refactor: Exclude JavaScript from toolchain task-I/O dispatch (#13540) (`115e850`) - refactor: Migrate change classification to immutable knowledge (#13543) (`5a55e8d`) - refactor: Separate prune rendering with golden coverage (#13546) (`62a047a`) - refactor: Delete JS format interpretation from prune orchestration (#13554) (`f1b37f6`) - refactor: Audit remaining JS knowledge consumer reads (#13556) (`7f12c3f`) - refactor: Migrate MFE dependency detection off PackageInfo (#13558) (`4cb9d94`) - refactor: Remove prune PackageInfo dependencies (#13562) (`1bb9233`) - refactor: Remove residual runtime PackageInfo gates (#13564) (`c8eaedc`) - refactor: Migrate boundary diagnostics off PackageInfo (#13571) (`f977db3`) - docs: Audit Cargo package knowledge (#13572) (`53dd13e`) - refactor: Complete Cargo relationship and resolution knowledge (#13576) (`b29e30c`) - refactor: Port Cargo task and contract knowledge (#13581) (`74446c5`) - refactor: Port Cargo watch and prune knowledge (#13582) (`acd3be1`) - refactor: Remove runtime toolchain dispatch (#13584) (`2c10201`) - ci: Restore Cargo target for lockfile tests (#13588) (`e848ad1`) - refactor: Replace toolchains with repository contributors (#13585) (`5d06e95`) - refactor: Remove Cargo contributor plumbing (#13586) (`6ea1daf`) - refactor: Remove ToolchainId runtime dispatch (#13587) (`24fbd3e`) - refactor: Route task behavior through contract domains (#13589) (`da88240`) - refactor: Route package consumers by manifest (#13591) (`bb12c72`) - refactor: Route MFE eligibility by manifest (#13592) (`08c7a74`) - refactor: Project manifest-derived repository facts (#13593) (`64e2a7e`) - refactor: Route residual task behavior by capability (#13595) (`fe9b72d`) - refactor: Route resolution through explicit domains (#13596) (`be21801`) - refactor: Route N-API package listing by manifest (#13597) (`8f44636`) - docs: Vercel Remote Cache authentication with OIDC policies (#13140) (`c84ed36`) - refactor: Own resolution fingerprints in repository (#13598) (`16708a9`) - refactor: Fail closed on invalid relationships (#13599) (`1a237de`) - perf: Reuse Cargo metadata discovery snapshot (#13600) (`2fa79c5`) - refactor: Resolve MFE package ownership from graph (#13601) (`0253836`) - refactor: Remove retained package payloads (#13603) (`58d9660`) - feat: Add native Cargo format task (#13606) (`ab15587`) - refactor: Compose repository graphs for optional toolchains (#13608) (`915e82b`) - ci: Invalidate Cap'n Proto caches (#13616) (`5cf35ad`) - feat: Discover uv workspaces (#13609) (`8715646`) - feat: Run native uv tasks (#13610) (`4195e41`) - feat: Hash uv lockfile closures (#13611) (`e14de24`) - fix: Make Windows Cap'n Proto cache relocatable (#13621) (`00538d0`) - feat: Watch uv workspace changes (#13612) (`b2e25d4`) - feat: Prune uv workspaces (#13613) (`f8f288e`) - fix: Fall back to polling on macOS (#13622) (`b3dc99b`) - test: Add uv workspace integration coverage (#13602) (`dd87718`) - chore: Release Turborepo 2.10.8 (#13626) (`adbfec7`) - perf: Walk literal-prefix tree globs without wax compilation (#13522) (`eb42f23`) - fix: Accept semver ranges in devEngines.packageManager.version (#13623) (`5297aa2`) - docs: Explain affected package invalidation reasons (#13594) (`c6fbc97`) - perf(lockfiles): Borrow field-name scalars in the pnpm fast parser (#13648) (`73e8d8c`) - perf(repository): Avoid discarded alias allocation in Relationship (#13650) (`b888891`) - perf(lockfiles): Drop redundant human_name clone for pnpm v7/v9 (#13649) (`2effc86`) - perf: Index workspace nodes by name in project_relationships (#13647) (`0bf6973`) - perf: Share resolution identity lists across identical workspace closures (#13641) (`5107207`) - docs: Fix duplicated word in runtime dependencies guide summary (#13630) (`0664de8`) - refactor: Remove turborepo-lsp dependency on turborepo-lib (#13631) (`8ff1ad7`) - perf: Index Bun nested lockfile entries by name for fallback resolution (#13633) (`c0a8996`) - perf: Memoize framework inference per package during task hashing (#13634) (`21ea1d0`) - perf: Avoid materializing transient declarations in external_dependencies (#13646) (`a892a89`) - perf: Enable shared closure DP for npm and yarn1 lockfiles (#13635) (`04db9a8`) - perf: Parse pnpm explicit-key entries in the lockfile fast path (#13640) (`83ae3d9`) - perf: Parallelize resolution fingerprint hashing (#13642) (`95f2297`) - perf: Build resolution identity lists in parallel (#13643) (`58e4e8a`) - perf: Intern resolution identities as Arc<str> across closures (#13645) (`6af5423`) - fix: Compose affected tasks with package filters (#13656) (`0b1f466`) - docs: Explain worktree cache path isolation (#13657) (`9e2865e`) - fix: Upgrade brace-expansion to 5.0.9 (#13658) (`e247a0e`) - docs: Correct verified inaccuracies in the Turborepo Agent Skill (#13644) (`c05ed3d`) - chore: Update Next.js to 16.3.0 (#13659) (`a936402`) - fix: don't use eprintln! in the panic hook (#13637) (`658fd54`) - fix: Invalidate only when Git ignore sources change (#13632) (`f2957a2`) - docs: Update Geistdocs to 1.19.4 (#13680) (`3617c78`) - docs: Exclude Turborepo from its own OSS products menu (#13681) (`43ee46a`) - docs: Use the geistdocs Turborepo logo in the navbar (#13682) (`d43eec5`) - docs: Update redirected vercel.com/nextjs.org links to current targets (#13685) (`b23e283`) - refactor: Generalize native command arguments (#13664) (`e797251`) - refactor: Move native contracts to tasks (#13665) (`851857d`) - docs: Fix loadTransformers reference in turbo-codemod README (#13683) (`7b8144e`) - refactor: Model native task execution explicitly (#13666) (`2634f3c`) - feat: Compose aggregate native task dependencies (#13667) (`ef8b3f3`) - fix: Respect aggregate task overrides (#13668) (`81f88f2`) - test: Stabilize watch task inputs regression test (#13686) (`308ea6b`) - feat: Parse Python quality tool declarations (#13669) (`b1d5dc5`) - feat: Resolve Python quality plans (#13670) (`439b465`) - refactor: Extract uv native task specs (#13671) (`e14f04e`) - feat: Synthesize Python quality tasks (#13672) (`94708ad`) - test: Cover Python quality task commands (#13673) (`0d43ff3`) - feat: Hash Python quality task inputs (#13674) (`3584a5f`) - test: Cover Python quality task graph (#13675) (`09bd548`) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew
pushed a commit
that referenced
this pull request
Aug 7, 2026
## Release v2.10.9 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/31216733075) ### Changes - chore: Release Turborepo 2.10.8 (#13626) (`adbfec7`) - perf: Walk literal-prefix tree globs without wax compilation (#13522) (`eb42f23`) - fix: Accept semver ranges in devEngines.packageManager.version (#13623) (`5297aa2`) - docs: Explain affected package invalidation reasons (#13594) (`c6fbc97`) - perf(lockfiles): Borrow field-name scalars in the pnpm fast parser (#13648) (`73e8d8c`) - perf(repository): Avoid discarded alias allocation in Relationship (#13650) (`b888891`) - perf(lockfiles): Drop redundant human_name clone for pnpm v7/v9 (#13649) (`2effc86`) - perf: Index workspace nodes by name in project_relationships (#13647) (`0bf6973`) - perf: Share resolution identity lists across identical workspace closures (#13641) (`5107207`) - docs: Fix duplicated word in runtime dependencies guide summary (#13630) (`0664de8`) - refactor: Remove turborepo-lsp dependency on turborepo-lib (#13631) (`8ff1ad7`) - perf: Index Bun nested lockfile entries by name for fallback resolution (#13633) (`c0a8996`) - perf: Memoize framework inference per package during task hashing (#13634) (`21ea1d0`) - perf: Avoid materializing transient declarations in external_dependencies (#13646) (`a892a89`) - perf: Enable shared closure DP for npm and yarn1 lockfiles (#13635) (`04db9a8`) - perf: Parse pnpm explicit-key entries in the lockfile fast path (#13640) (`83ae3d9`) - perf: Parallelize resolution fingerprint hashing (#13642) (`95f2297`) - perf: Build resolution identity lists in parallel (#13643) (`58e4e8a`) - perf: Intern resolution identities as Arc<str> across closures (#13645) (`6af5423`) - fix: Compose affected tasks with package filters (#13656) (`0b1f466`) - docs: Explain worktree cache path isolation (#13657) (`9e2865e`) - fix: Upgrade brace-expansion to 5.0.9 (#13658) (`e247a0e`) - docs: Correct verified inaccuracies in the Turborepo Agent Skill (#13644) (`c05ed3d`) - chore: Update Next.js to 16.3.0 (#13659) (`a936402`) - fix: don't use eprintln! in the panic hook (#13637) (`658fd54`) - fix: Invalidate only when Git ignore sources change (#13632) (`f2957a2`) - docs: Update Geistdocs to 1.19.4 (#13680) (`3617c78`) - docs: Exclude Turborepo from its own OSS products menu (#13681) (`43ee46a`) - docs: Use the geistdocs Turborepo logo in the navbar (#13682) (`d43eec5`) - docs: Update redirected vercel.com/nextjs.org links to current targets (#13685) (`b23e283`) - refactor: Generalize native command arguments (#13664) (`e797251`) - refactor: Move native contracts to tasks (#13665) (`851857d`) - docs: Fix loadTransformers reference in turbo-codemod README (#13683) (`7b8144e`) - refactor: Model native task execution explicitly (#13666) (`2634f3c`) - feat: Compose aggregate native task dependencies (#13667) (`ef8b3f3`) - fix: Respect aggregate task overrides (#13668) (`81f88f2`) - test: Stabilize watch task inputs regression test (#13686) (`308ea6b`) - feat: Parse Python quality tool declarations (#13669) (`b1d5dc5`) - feat: Resolve Python quality plans (#13670) (`439b465`) - refactor: Extract uv native task specs (#13671) (`e14f04e`) - feat: Synthesize Python quality tasks (#13672) (`94708ad`) - test: Cover Python quality task commands (#13673) (`0d43ff3`) - feat: Hash Python quality task inputs (#13674) (`3584a5f`) - test: Cover Python quality task graph (#13675) (`09bd548`) - chore: Release Turborepo 2.10.9-canary.1 (#13687) (`c09a92f`) - docs: Document dependency-driven Python tasks (#13676) (`a98e5cd`) - fix: Prune Bun wildcard workspace dev dependencies (#13694) (`efe4e1b`) - fix: Prevent Windows process cleanup PID reuse (#13695) (`3b0e57f`) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The shared transitive-closure DP (
closure_dp.rs) computes each workspace's closure once over the global package graph, but only pnpm implementedtransitive_edge_resolver— npm and yarn1 always fell back to the legacy walk, which re-resolves and re-visits every edge of the shared dependency graph once per workspace (the resolve cache is keyed by workspace, so nothing is shared).Both formats can prove per-edge workspace independence cheaply, which is all the DP requires:
all_dependencies, which only emits keys present in the packages map (find_dep_in_lockfilewalks thenode_moduleshierarchy and returns real keys), andresolve_packagematches such a key with its first candidate, never consulting the workspace. The resolver mirrors that key-hit branch exactly and conservatively reports any non-key name asWorkspaceSensitive(defensive; unreachable viaall_dependenciesoutput). Workspace-nested versions (apps/web/node_modules/foo) are unaffected: direct dependencies are still resolved per workspace by the DP's step 1.resolve_packageignores its workspace argument entirely, so every edge is globally uniform by construction.Benchmarks (synthetic lockfile, 50 workspaces sharing a 4,500-package graph, release build, best of 3):
The gap grows with workspace count since the legacy walk is O(workspaces × closure size) while the DP is one graph pass plus bitset unions. This runs on every
turbo runstartup as part of package-graph construction.Testing Instructions
cargo test -p turborepo-lockfiles— 283 tests pass, including two new differential tests inclosure_dp.rsasserting the DP result exactly matches the legacy walk (the independent oracle): an npm case with a workspace-nested package version plus a deeply nested transitive dep, and a yarn1 multi-workspace case.WorkspaceSensitiveedge still aborts the DP and falls back to the legacy walk, so behavior is preserved unconditionally — the DP is only ever a faster path to the identical answer.cargo clippy -p turborepo-lockfiles --all-targetsis clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU
Generated by Claude Code