perf: Parse pnpm explicit-key entries in the lockfile fast path by anthonyshew · Pull Request #13640 · vercel/turborepo

anthonyshew · 2026-08-02T15:39:29Z

Description

Profiling package-graph construction on a large real-world pnpm monorepo showed parse_lockfile dominating the critical path: the structural fast parser was declining the lockfile and falling back to the general YAML parser.

The cause: the yaml library pnpm uses emits explicit keys (? key with the value on the following : value line) for mapping keys longer than 1024 bytes — the YAML spec limit for implicit keys. Heavily peer-suffixed snapshot keys hit this in practice, and a single such entry (the profiled repo had two, in a 75k-line lockfile) knocked the entire file off the fast path.

Changes:

Line scanner (data_scanner.rs): a ? key line records a pending explicit value; the next content line must be its : value line at the same indent, carrying either an inline scalar or a block mapping whose first entry starts on that line (: dependencies: — the shape pnpm emits). Any other shape (orphaned ? key, wrong indent, multi-line quoted key) still declines to the general parser, preserving the tier's "never accept input with a different meaning than saphyr" contract.
Chunked parallel splitter (data_fast_parse.rs): ? lines are now child-entry starts and : lines interior, so a chunk boundary can never separate a ? /: pair. Also hardened: the splitter now declines when non-blank content precedes the first child entry, since the chunk assembly silently drops everything before the first start (previously a latent accept-with-drop edge at ≥256 KB fragment sizes).

Measured on the real repo (75k-line lockfile, release build, steady-state medians over 12 samples):

Phase	Before	After
`parse_lockfile`	~139 ms	~29 ms (4.8×)
package graph build (total)	~240 ms	~125 ms (1.9×)

This cost is paid on every turbo invocation in affected repos. The fast-parse result was verified byte-identical to the serde oracle (including re-encoding) on two large production lockfiles (165k and 75k lines).

Testing Instructions

cargo test -p turborepo-lockfiles — 285 tests pass. New differential tests cover: explicit keys with block-mapping and inline-scalar values at top-level and nested depths (test_explicit_keys_differential), decline cases (test_explicit_key_without_value_line_falls_back), a >512 KB input exercising the chunked splitter with explicit keys at start/middle/end (test_chunked_split_handles_explicit_keys), and the content-before-first-entry hardening (test_chunked_split_rejects_content_before_first_entry).
All differential tests assert scanner == parallel-split == serde on accepted inputs, and that declined inputs are declined by every tier.
cargo clippy -p turborepo-lockfiles --all-targets is clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU

Generated by Claude Code

The yaml library pnpm uses emits explicit keys (`? key` with the value on the following `: value` line) for mapping keys longer than 1024 bytes, the YAML limit for implicit keys. Real monorepos hit this with heavily peer-suffixed snapshot keys, and a single such entry knocked the entire lockfile off the structural fast path onto the general YAML parser. Teach the line scanner the explicit-key form: a `? key` line records a pending explicit value, and the next content line must be its `: value` line at the same indent, carrying either an inline scalar or a block mapping whose first entry starts on that line. Anything else still declines to the general parser. The chunked parallel splitter now treats `? ` lines as child-entry starts and `: ` lines as interior, so a chunk boundary can never separate the pair; it also declines when non-blank content precedes the first child entry, since the chunk assembly would silently drop it. New differential tests cover both value shapes at multiple depths, decline cases (orphaned `? key`, wrong-indent `: value`), and a chunked-scale input with explicit keys at the start, middle, and end. On a large real-world monorepo whose lockfile (75k lines) contains two such keys, lockfile parse drops from ~139ms to ~29ms (4.8x) and total package graph construction from ~240ms to ~125ms, on every turbo invocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VGZtfhEfgWhrAFMTMhwTfU

vercel · 2026-08-02T15:39:33Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
examples-basic-web	Ready	Preview	Aug 2, 2026 3:40pm
examples-designsystem-docs	Ready	Preview	Aug 2, 2026 3:40pm
examples-gatsby-web	Ready	Preview	Aug 2, 2026 3:40pm
examples-kitchensink-blog	Ready	Preview	Aug 2, 2026 3:40pm
examples-nonmonorepo	Ready	Preview	Aug 2, 2026 3:40pm
examples-svelte-web	Ready	Preview	Aug 2, 2026 3:40pm
examples-tailwind-web	Ready	Preview	Aug 2, 2026 3:40pm
examples-vite-web	Ready	Preview	Aug 2, 2026 3:40pm
turbo-site	Ready	Preview	Aug 2, 2026 3:40pm
turborepo-eve-agent	Ready	Preview	Aug 2, 2026 3:40pm

## Release v2.10.9-canary.1 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/31044275876) ### Changes - refactor: Delete legacy external-resolution PackageInfo state (#13526) (`fbe88d3`) - chore: Release Turborepo 2.10.8-canary.4 (#13557) (`887d9b1`) - refactor: Remove external declaration compatibility paths (#13527) (`3c8448d`) - refactor: Produce immutable native task and command knowledge (#13528) (`18d0bc3`) - refactor: Migrate native task registration and suggestions (#13529) (`7b5ee63`) - refactor: Migrate turbo-json native task synthesis (#13530) (`2d004f0`) - refactor: Migrate persistent and recursive task validation (#13531) (`dd7519b`) - refactor: Migrate native task definition precedence (#13532) (`8792ce9`) - refactor: Migrate engine native command planning (#13533) (`67e4e9e`) - refactor: Migrate executor native command resolution (#13534) (`1b8accc`) - refactor: Migrate native task query, devtools, and LSP views (#13535) (`2753b76`) - refactor: Migrate command summaries and delete legacy task paths (#13536) (`adbad50`) - refactor: Produce immutable task-contract knowledge (#13537) (`97bff31`) - refactor: Migrate engine task-contract composition (#13538) (`fab50c5`) - refactor: Migrate hashing engines to task contracts (#13539) (`76206f8`) - refactor: Exclude JavaScript from toolchain task-I/O dispatch (#13540) (`115e850`) - refactor: Migrate change classification to immutable knowledge (#13543) (`5a55e8d`) - refactor: Separate prune rendering with golden coverage (#13546) (`62a047a`) - refactor: Delete JS format interpretation from prune orchestration (#13554) (`f1b37f6`) - refactor: Audit remaining JS knowledge consumer reads (#13556) (`7f12c3f`) - refactor: Migrate MFE dependency detection off PackageInfo (#13558) (`4cb9d94`) - refactor: Remove prune PackageInfo dependencies (#13562) (`1bb9233`) - refactor: Remove residual runtime PackageInfo gates (#13564) (`c8eaedc`) - refactor: Migrate boundary diagnostics off PackageInfo (#13571) (`f977db3`) - docs: Audit Cargo package knowledge (#13572) (`53dd13e`) - refactor: Complete Cargo relationship and resolution knowledge (#13576) (`b29e30c`) - refactor: Port Cargo task and contract knowledge (#13581) (`74446c5`) - refactor: Port Cargo watch and prune knowledge (#13582) (`acd3be1`) - refactor: Remove runtime toolchain dispatch (#13584) (`2c10201`) - ci: Restore Cargo target for lockfile tests (#13588) (`e848ad1`) - refactor: Replace toolchains with repository contributors (#13585) (`5d06e95`) - refactor: Remove Cargo contributor plumbing (#13586) (`6ea1daf`) - refactor: Remove ToolchainId runtime dispatch (#13587) (`24fbd3e`) - refactor: Route task behavior through contract domains (#13589) (`da88240`) - refactor: Route package consumers by manifest (#13591) (`bb12c72`) - refactor: Route MFE eligibility by manifest (#13592) (`08c7a74`) - refactor: Project manifest-derived repository facts (#13593) (`64e2a7e`) - refactor: Route residual task behavior by capability (#13595) (`fe9b72d`) - refactor: Route resolution through explicit domains (#13596) (`be21801`) - refactor: Route N-API package listing by manifest (#13597) (`8f44636`) - docs: Vercel Remote Cache authentication with OIDC policies (#13140) (`c84ed36`) - refactor: Own resolution fingerprints in repository (#13598) (`16708a9`) - refactor: Fail closed on invalid relationships (#13599) (`1a237de`) - perf: Reuse Cargo metadata discovery snapshot (#13600) (`2fa79c5`) - refactor: Resolve MFE package ownership from graph (#13601) (`0253836`) - refactor: Remove retained package payloads (#13603) (`58d9660`) - feat: Add native Cargo format task (#13606) (`ab15587`) - refactor: Compose repository graphs for optional toolchains (#13608) (`915e82b`) - ci: Invalidate Cap'n Proto caches (#13616) (`5cf35ad`) - feat: Discover uv workspaces (#13609) (`8715646`) - feat: Run native uv tasks (#13610) (`4195e41`) - feat: Hash uv lockfile closures (#13611) (`e14de24`) - fix: Make Windows Cap'n Proto cache relocatable (#13621) (`00538d0`) - feat: Watch uv workspace changes (#13612) (`b2e25d4`) - feat: Prune uv workspaces (#13613) (`f8f288e`) - fix: Fall back to polling on macOS (#13622) (`b3dc99b`) - test: Add uv workspace integration coverage (#13602) (`dd87718`) - chore: Release Turborepo 2.10.8 (#13626) (`adbfec7`) - perf: Walk literal-prefix tree globs without wax compilation (#13522) (`eb42f23`) - fix: Accept semver ranges in devEngines.packageManager.version (#13623) (`5297aa2`) - docs: Explain affected package invalidation reasons (#13594) (`c6fbc97`) - perf(lockfiles): Borrow field-name scalars in the pnpm fast parser (#13648) (`73e8d8c`) - perf(repository): Avoid discarded alias allocation in Relationship (#13650) (`b888891`) - perf(lockfiles): Drop redundant human_name clone for pnpm v7/v9 (#13649) (`2effc86`) - perf: Index workspace nodes by name in project_relationships (#13647) (`0bf6973`) - perf: Share resolution identity lists across identical workspace closures (#13641) (`5107207`) - docs: Fix duplicated word in runtime dependencies guide summary (#13630) (`0664de8`) - refactor: Remove turborepo-lsp dependency on turborepo-lib (#13631) (`8ff1ad7`) - perf: Index Bun nested lockfile entries by name for fallback resolution (#13633) (`c0a8996`) - perf: Memoize framework inference per package during task hashing (#13634) (`21ea1d0`) - perf: Avoid materializing transient declarations in external_dependencies (#13646) (`a892a89`) - perf: Enable shared closure DP for npm and yarn1 lockfiles (#13635) (`04db9a8`) - perf: Parse pnpm explicit-key entries in the lockfile fast path (#13640) (`83ae3d9`) - perf: Parallelize resolution fingerprint hashing (#13642) (`95f2297`) - perf: Build resolution identity lists in parallel (#13643) (`58e4e8a`) - perf: Intern resolution identities as Arc<str> across closures (#13645) (`6af5423`) - fix: Compose affected tasks with package filters (#13656) (`0b1f466`) - docs: Explain worktree cache path isolation (#13657) (`9e2865e`) - fix: Upgrade brace-expansion to 5.0.9 (#13658) (`e247a0e`) - docs: Correct verified inaccuracies in the Turborepo Agent Skill (#13644) (`c05ed3d`) - chore: Update Next.js to 16.3.0 (#13659) (`a936402`) - fix: don't use eprintln! in the panic hook (#13637) (`658fd54`) - fix: Invalidate only when Git ignore sources change (#13632) (`f2957a2`) - docs: Update Geistdocs to 1.19.4 (#13680) (`3617c78`) - docs: Exclude Turborepo from its own OSS products menu (#13681) (`43ee46a`) - docs: Use the geistdocs Turborepo logo in the navbar (#13682) (`d43eec5`) - docs: Update redirected vercel.com/nextjs.org links to current targets (#13685) (`b23e283`) - refactor: Generalize native command arguments (#13664) (`e797251`) - refactor: Move native contracts to tasks (#13665) (`851857d`) - docs: Fix loadTransformers reference in turbo-codemod README (#13683) (`7b8144e`) - refactor: Model native task execution explicitly (#13666) (`2634f3c`) - feat: Compose aggregate native task dependencies (#13667) (`ef8b3f3`) - fix: Respect aggregate task overrides (#13668) (`81f88f2`) - test: Stabilize watch task inputs regression test (#13686) (`308ea6b`) - feat: Parse Python quality tool declarations (#13669) (`b1d5dc5`) - feat: Resolve Python quality plans (#13670) (`439b465`) - refactor: Extract uv native task specs (#13671) (`e14f04e`) - feat: Synthesize Python quality tasks (#13672) (`94708ad`) - test: Cover Python quality task commands (#13673) (`0d43ff3`) - feat: Hash Python quality task inputs (#13674) (`3584a5f`) - test: Cover Python quality task graph (#13675) (`09bd548`) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

## Release v2.10.9 > [!CAUTION] > Versioned docs aliasing FAILED. [View logs](https://github.com/vercel/turborepo/actions/runs/31216733075) ### Changes - chore: Release Turborepo 2.10.8 (#13626) (`adbfec7`) - perf: Walk literal-prefix tree globs without wax compilation (#13522) (`eb42f23`) - fix: Accept semver ranges in devEngines.packageManager.version (#13623) (`5297aa2`) - docs: Explain affected package invalidation reasons (#13594) (`c6fbc97`) - perf(lockfiles): Borrow field-name scalars in the pnpm fast parser (#13648) (`73e8d8c`) - perf(repository): Avoid discarded alias allocation in Relationship (#13650) (`b888891`) - perf(lockfiles): Drop redundant human_name clone for pnpm v7/v9 (#13649) (`2effc86`) - perf: Index workspace nodes by name in project_relationships (#13647) (`0bf6973`) - perf: Share resolution identity lists across identical workspace closures (#13641) (`5107207`) - docs: Fix duplicated word in runtime dependencies guide summary (#13630) (`0664de8`) - refactor: Remove turborepo-lsp dependency on turborepo-lib (#13631) (`8ff1ad7`) - perf: Index Bun nested lockfile entries by name for fallback resolution (#13633) (`c0a8996`) - perf: Memoize framework inference per package during task hashing (#13634) (`21ea1d0`) - perf: Avoid materializing transient declarations in external_dependencies (#13646) (`a892a89`) - perf: Enable shared closure DP for npm and yarn1 lockfiles (#13635) (`04db9a8`) - perf: Parse pnpm explicit-key entries in the lockfile fast path (#13640) (`83ae3d9`) - perf: Parallelize resolution fingerprint hashing (#13642) (`95f2297`) - perf: Build resolution identity lists in parallel (#13643) (`58e4e8a`) - perf: Intern resolution identities as Arc<str> across closures (#13645) (`6af5423`) - fix: Compose affected tasks with package filters (#13656) (`0b1f466`) - docs: Explain worktree cache path isolation (#13657) (`9e2865e`) - fix: Upgrade brace-expansion to 5.0.9 (#13658) (`e247a0e`) - docs: Correct verified inaccuracies in the Turborepo Agent Skill (#13644) (`c05ed3d`) - chore: Update Next.js to 16.3.0 (#13659) (`a936402`) - fix: don't use eprintln! in the panic hook (#13637) (`658fd54`) - fix: Invalidate only when Git ignore sources change (#13632) (`f2957a2`) - docs: Update Geistdocs to 1.19.4 (#13680) (`3617c78`) - docs: Exclude Turborepo from its own OSS products menu (#13681) (`43ee46a`) - docs: Use the geistdocs Turborepo logo in the navbar (#13682) (`d43eec5`) - docs: Update redirected vercel.com/nextjs.org links to current targets (#13685) (`b23e283`) - refactor: Generalize native command arguments (#13664) (`e797251`) - refactor: Move native contracts to tasks (#13665) (`851857d`) - docs: Fix loadTransformers reference in turbo-codemod README (#13683) (`7b8144e`) - refactor: Model native task execution explicitly (#13666) (`2634f3c`) - feat: Compose aggregate native task dependencies (#13667) (`ef8b3f3`) - fix: Respect aggregate task overrides (#13668) (`81f88f2`) - test: Stabilize watch task inputs regression test (#13686) (`308ea6b`) - feat: Parse Python quality tool declarations (#13669) (`b1d5dc5`) - feat: Resolve Python quality plans (#13670) (`439b465`) - refactor: Extract uv native task specs (#13671) (`e14f04e`) - feat: Synthesize Python quality tasks (#13672) (`94708ad`) - test: Cover Python quality task commands (#13673) (`0d43ff3`) - feat: Hash Python quality task inputs (#13674) (`3584a5f`) - test: Cover Python quality task graph (#13675) (`09bd548`) - chore: Release Turborepo 2.10.9-canary.1 (#13687) (`c09a92f`) - docs: Document dependency-driven Python tasks (#13676) (`a98e5cd`) - fix: Prune Bun wildcard workspace dev dependencies (#13694) (`efe4e1b`) - fix: Prevent Windows process cleanup PID reuse (#13695) (`3b0e57f`) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

anthonyshew requested review from a team and tknickman August 2, 2026 15:39

vercel Bot deployed to Preview – turborepo-eve-agent August 2, 2026 15:39 View deployment

tknickman approved these changes Aug 3, 2026

View reviewed changes

anthonyshew merged commit 83ae3d9 into main Aug 3, 2026
72 of 74 checks passed

anthonyshew deleted the claude/perf-pnpm-explicit-keys branch August 3, 2026 21:37

github-actions Bot mentioned this pull request Aug 5, 2026

chore: Release Turborepo 2.10.9-canary.1 #13687

Merged

github-actions Bot mentioned this pull request Aug 7, 2026

chore: Release Turborepo 2.10.9 #13696

Merged

Navigation Menu

Uh oh!

perf: Parse pnpm explicit-key entries in the lockfile fast path - #13640

perf: Parse pnpm explicit-key entries in the lockfile fast path#13640
anthonyshew merged 1 commit into
mainfrom
claude/perf-pnpm-explicit-keys

anthonyshew commented Aug 2, 2026

Uh oh!

vercel Bot commented Aug 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

anthonyshew commented Aug 2, 2026

Description

Testing Instructions

Uh oh!

vercel Bot commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vercel Bot commented Aug 2, 2026 •

edited

Loading