feat: project dependencies and workspaces#641
Draft
lwshang wants to merge 15 commits into
Draft
Conversation
Introduce a top-level `dependencies:` block in icp.yaml for declaring another icp project this project depends on. Each entry has a local alias (`name`), a `path` to the dependency's project dir, and an optional `canisters` exposure subset (omit = expose all). This commit adds only the manifest type + schema; consolidation and deploy behavior follow in later commits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ject Extend consolidate_manifest to import declared dependency projects: - Recursively load each dependency's icp.yaml and import ALL of its canisters (deploy-all), keyed by their app-root-relative path (e.g. `umbrella/openemail:backend`). The `canisters` field is an exposure filter for env vars only, not a deployment filter. - De-duplicate instances by canonicalized directory, so a diamond dependency (e.g. two sibling submodules both depending on `../openemail`) deploys once. - Detect dependency cycles. - Compute per-project `PUBLIC_CANISTER_ID` bindings on each Canister so a dependency's canisters keep their standalone view (own siblings by bare name) while the parent sees only the exposed dependency canisters under `<alias>:`. - Reserve `:` in canister names and dependency aliases; validate alias uniqueness and alias/canister collisions. - Translate dependency canisters' name-based controller refs to store keys. The canonical `Canister` gains a `bindings` map consumed by the deploy step (next commit). For a project with no dependencies, bindings map every canister to itself, preserving today's flat env-var behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…store namespaced-name safe - set_binding_env_vars_many now derives each canister's PUBLIC_CANISTER_ID env vars from its own `bindings` map instead of a single flat set, so a dependency's canisters keep the view their project expects and the parent sees only the exposed dependency canisters. Bindings are filtered to ids present in the environment; a dependency-free project is unaffected (every canister still sees every sibling). - The artifact store percent-encodes `/`, `\`, `:`, `%` in canister names so namespaced dependency canister keys (e.g. `vendor/openemail:backend`) are valid filenames on all platforms. Plain names are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A dependency canister's namespaced name (e.g. `dep:backend`, `vendor/dep:backend`) is not a valid DNS label, so it cannot get a friendly `<name>.<env>.<domain>` subdomain. Skip the friendly host for such names and use the principal host, avoiding a malformed/incorrect URL when printing deployed canister URLs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add examples/icp-project-dependency demonstrating an app that vendors an `openemail` dependency and exposes its `backend` canister. - Add an integration test that deploys an app + dependency to a managed network and asserts the app sees PUBLIC_CANISTER_ID:openemail:backend while the dependency's canisters keep their standalone view. - Document dependency projects in the canister-discovery guide and CHANGELOG. - Move the store_artifact test module to the end of the file (clippy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ndencies Cover the layout where two independent sub-projects (service-a, service-b) each depend on the same sibling `openemail` via `../openemail`, and an app depends on both services. - Add examples/icp-project-dependency-shared demonstrating the umbrella layout. - Add an integration test that deploys the app + both services + openemail to a managed network and asserts openemail is deployed once and both services' `PUBLIC_CANISTER_ID:openemail:backend` resolve to the same shared canister id. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ember Running an icp command inside a vendored sub-project now climbs to the workspace root — the top-most project that transitively declares the one you are in as a dependency — so the store, artifacts and network all come from that single root (single source-of-truth canister IDs). Resolution uses an early-stop, transitive-containment climb: an ancestor is adopted only if its dependency closure declares the starting project; the walk stops at the first ancestor that does not, so it never crosses a gap or adopts an unrelated ancestor. With no declaring ancestor it returns the nearest icp.yaml (today's behavior), so standalone projects are unaffected. Also exposes the project-root override as the ICP_PROJECT_ROOT env var, matching ICP_ENVIRONMENT / ICP_NETWORK; it forces the root with no climb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sub-project When `icp deploy` is run with no canister names from inside a vendored member (not the workspace root), it now defaults to deploying only that member's canisters — those whose directory is within the member — instead of the whole workspace, and announces the resolved workspace root so the upward resolution is visible. Deploying from the root (or a standalone project) is unchanged. Because resolution returns the workspace root, the member's ids are written to the single root store, so a from-member redeploy reuses ids and keeps cross-member env-var wiring valid. - ProjectRootLocate gains locate_member() (nearest icp.yaml at/above cwd, no climb; equals the root at the root or standalone). - ProjectLoad gains member_dir() (default None; ProjectLoadImpl derives it from the locator, Lazy forwards it) so mocks are unaffected. - project::member_scoped_canisters() computes the default target set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…clare targeted envs Deploying a vendored member to environment E now honors the member's own same-named environment (standalone-equivalence, §16.7): its per-canister settings/init_args are folded into the root's E, remapped to namespaced store keys, beneath any explicit root override. Precedence is root-override > member-env > canister-base. The member's env network binding and canister selection are ignored (network + store come from the root; the full member set is deployed). A member's overrides apply to its own canisters only; keys naming its dependencies are left to those dependencies. Strict rule: every vendored member must declare each environment the workspace targets. Missing members are recorded per-environment during consolidation and enforced lazily in get_environment when that environment is selected — so a member missing some other environment never blocks deploys to the ones it does declare — via GetEnvironmentError::MissingDependencyEnvironment. - project.rs: capture member environments during import; build_environment_canisters helper applies member-then-root overrides for explicit and implicit envs. - Project gains member_missing_envs. - Integration/example fixtures: dependencies now declare the targeted environment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The workspace-resolution/env-merge comments and test fixtures referenced section numbers (§16.x) of an external design document that is not part of the repository. Replace them with self-contained prose so the code does not point at a document readers cannot find. Comment/string-only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a dedicated concepts page covering the project-dependencies feature: declaring a dependency, deploy-all vs. exposure, per-project canister-ID injection and namespacing, the workspace model (running commands inside a vendored member resolves up to the workspace root for single source-of-truth IDs), member-scoped deploy, ICP_PROJECT_ROOT / --project-root-override, environment merge and the strict rule that members declare targeted environments, shared-dependency de-duplication, and self-containment. Register it in the sidebar and concepts index, and trim the overlapping "Dependency Projects" subsection in canister-discovery.md to a short pointer (DRY: the injection mechanism stays there; the feature lives on the new page). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Trim the Unreleased entry to the headline behavior (declare a dependency, deploy it alongside your project with injected IDs, run from a sub-project to resolve to the workspace root) and defer specifics to the new Project Dependencies concept guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lwshang
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a top-level
dependencies:block so a project can depend on anothericpproject vendored into it (typically as a git submodule), and makes vendored projects behave as a workspace: runningicpfrom inside a sub-project resolves up to the outermost project that declares it, so the whole workspace shares one network and one set of canister IDs.Full behavior — declaration, deploy, ID injection, member-scoped deploy, environments, de-duplication — is in the new Project Dependencies concept guide. This description focuses on the why behind the load-bearing decisions.
Why (key decisions)
Running from a sub-project resolves to the workspace root. The alternative — each sub-project acting standalone — meant
cd vendor/openemail && icp deploydeployed to a separate network and ID set than the parent, silently forking the very dependency you were iterating on. Resolving up to the root gives a single source of truth for canister IDs, so a from-member redeploy reuses IDs and keeps cross-project wiring valid.Resolution inverts the
dependencies:edges rather than using a workspace marker. A project's "root-ness" is contextual:openemailis its own root when cloned alone, but a member when cloned underapp. A staticworkspace: truemarker can't express that. Instead an ancestor is adopted only if it transitively declares the project you are in — this is "top-most" but bounded: an unrelatedicp.yamlhigher up never captures you, a dependency still works when cloned on its own, and the child never has to name its parent (stays self-contained).A dependency is always deployed in full;
canisters:only controls exposure. A dependency's canisters may call each other and icp-cli keeps no intra-project "requires" graph, so it can't safely prune.canisters:filters which IDs are exposed to the parent, not what deploys.Path-based store keys + directory-identity de-dup. Two projects can define the same canister name, so imported canisters are keyed by path relative to the root (
vendor/openemail:backend) — order-independent and portable. Identity is the resolved directory, so an "umbrella" diamond (two services vendoring../openemail) deploysopenemailonce.A member's own environment config is honored, but the workspace stays authoritative. Precedence is root override > member's env config > base: vendored code gets its author's per-environment settings (standalone-equivalence) while the root owns the network and ID store. Members must declare each environment the workspace targets — chosen over a silent fallback to catch typos and keep semantics unambiguous, and enforced lazily per selected environment so a member missing
stagingnever blocksdeploy -e local.Testing
Unit tests cover root resolution (clone scenarios, the umbrella diamond, gap early-stop, unrelated-ancestor rejection), member-scoping, env-merge precedence, the missing-environment rule, and every validation error. Integration tests deploy to a managed network for the single-dependency, shared/umbrella, and from-member workflows (asserting member IDs land in the root store). Full workspace suite green;
fmt/clippyclean.Breaking change
:is now reserved in canister names and dependency aliases as the namespace separator. A project with:in a canister name must rename it.Not in scope (design leaves room)
Pinned / already-deployed dependencies (additive per-network
pin:), candid/binding generation (each canister sources its own bindings), and per-canisterdepends_on.🤖 Generated with Claude Code