Skip to content

added default.nix for building reproducible dev environment#3479

Draft
pdg137 wants to merge 1 commit into
online-go:mainfrom
pdg137:nix
Draft

added default.nix for building reproducible dev environment#3479
pdg137 wants to merge 1 commit into
online-go:mainfrom
pdg137:nix

Conversation

@pdg137

@pdg137 pdg137 commented Apr 11, 2026

Copy link
Copy Markdown
Contributor

This PR proposes adding a default.nix to the top level, using the Nix build system to define a precise dev environment. This can be useful for developers familiar with Nix and also casual/infrequent contributors since (after Nix is installed) they only have to run one command to enter a dev shell:

nix-shell

No more reading docs to figure out exactly what you need to install (and what versions) to get it going! I don't know much about NodeJS environments so for me at least this satisfies a desire to keep them as encapsulated and well-defined as possible.

It works like this:

  1. The standard fetchYarnDeps function downloads the packages specified in yarn.lock and stores them in a cache folder in the Nix store. This cache is locked with a hash that will need to be updated from time to time, when yarn.lock and package.json change.
  2. The script builds a node_modules folder in --offline mode, using yarn.lock, package.json, and the cache as inputs.
  3. Finally a shell starts, running a startup hook that links node_modules into the right place.

The node_modules directory is read-only, which is a benefit since it's not possible for files in there to be corrupted. This required a change to Vite configuration to make it save its cache to the top level .vite instead of there. I made this configurable with an environment variable, so it still defaults to the standard location.

@github-actions

Copy link
Copy Markdown

Thanks for adding Nix support — this is a nice convenience for contributors who use Nix. Two issues to fix before this is ready:


node_modules/esbuild/install.js does not exist in esbuild v0.27.x

The build script runs:

${node}/bin/node node_modules/esbuild/install.js

But this install script was removed in older esbuild versions. Since v0.11+, esbuild ships its platform binary via optional peer packages (e.g. @esbuild/linux-x64), which yarn installs as normal dependencies. There is no install.js in esbuild 0.27.2 (the version pinned in yarn.lock). This line will fail with a module-not-found error and break every Nix build. It can simply be removed — the esbuild binary is already installed by yarn install --offline through the optional dependency.


Hardcoded unrs-resolver version will break on the next dependency update

${node}/bin/node ../napi-postinstall/lib/cli.js unrs-resolver 1.11.1 check

The version 1.11.1 is hardcoded here but yarn.lock pins unrs-resolver to ^1.7.11 — it will advance independently. When yarn.lock is updated to a newer patch/minor, this hardcoded version will be wrong and the post-install check will fail. A more robust approach is to read the version from the installed package itself:

(
  cd node_modules/unrs-resolver
  UNRS_VERSION=$(${node}/bin/node -e "process.stdout.write(require('./package.json').version)")
  ${node}/bin/node ../napi-postinstall/lib/cli.js unrs-resolver "$UNRS_VERSION" check
)

@github-actions

Copy link
Copy Markdown

Review notes

default.nix — fragile native-addon postinstall

The buildPhase manually replays the install script for one specific package:

(
  cd node_modules/unrs-resolver
  UNRS_VERSION=$(${node}/bin/node -e "process.stdout.write(require('./package.json').version)")
  ${node}/bin/node ../napi-postinstall/lib/cli.js unrs-resolver "$UNRS_VERSION" check
)

unrs-resolver is currently the only dependency in this tree that uses napi-postinstall, so this works today. The risk is that future yarn.lock updates could bring in additional native addons with the same pattern. Because --ignore-scripts suppresses all postinstall hooks, those new packages would silently end up without their native binary, causing cryptic runtime failures rather than a clear build-time error.

One improvement would be to derive the list of packages needing napi-postinstall dynamically — e.g. by scanning node_modules/*/package.json for scripts.postinstall referencing napi-postinstall — rather than hardcoding the package name. Alternatively, a comment noting that this list must be kept in sync with napi-postinstall-using deps would at least surface the maintenance requirement when reviewing dependency bumps.


The vite.config.ts change and .gitignore addition look fine.

@github-actions

Copy link
Copy Markdown

Code Review

ln -sfT is not portable to macOS

In the shellHook (line ~100 of default.nix):

ln -sfT ${node-modules}/node_modules ./node_modules

The -T flag is a GNU coreutils extension and is not available on macOS's BSD ln. Since Nix is widely used on macOS via nix-darwin, this will fail for macOS users unless pkgs.coreutils is added to buildInputs. A portable alternative would be:

rm -f ./node_modules
ln -s ${node-modules}/node_modules ./node_modules

Or add pkgs.coreutils to buildInputs to ensure GNU ln is available.


--ignore-scripts with manual install script replay is fragile

The build phase skips all install scripts (--ignore-scripts) and then manually replays only the unrs-resolver post-install step. The comment in the file acknowledges this explicitly:

"This must be kept in sync with new packages that need to run install scripts."

If any new native package dependency is added to package.json (e.g., a new esbuild version, a rollup native binding, or another NAPI module), the Nix build will silently produce a broken node_modules rather than failing at build time. The failure will only surface later as a confusing runtime error.

It would be worth either (a) documenting this as a known limitation in a contributing guide for Nix users, or (b) exploring whether pkgs.yarn2nix or a newer Nix tool can handle install scripts more robustly.


The rest of the changes (vite cacheDir, .gitignore) look correct.

@github-actions

Copy link
Copy Markdown

Code Review

napi-postinstall-alias will fail for scoped packages

The alias script uses a relative path to find the napi-postinstall CLI:

napi-postinstall-alias = pkgs.writeShellScriptBin "napi-postinstall"
  "exec ${node}/bin/node ../napi-postinstall/lib/cli.js $@";

Yarn sets CWD to the package directory when running postinstall scripts. For an unscoped package like node_modules/unrs-resolver, ../napi-postinstall correctly resolves to node_modules/napi-postinstall. However, for any scoped package at node_modules/@scope/some-package, .. resolves to node_modules/@scope/ — the wrong directory.

This works today because unrs-resolver is unscoped, but would silently break if a future dependency is scoped and uses napi-postinstall. A more robust path would avoid the relative traversal, for example by searching upward for node_modules or hard-coding the derivation path after node-modules is built. At minimum, a comment noting the constraint would help.


Hard dependency on a personal GitHub repository

mkBuildableShell is fetched from the PR author's personal GitHub account (pdg137/mkBuildableShell). The content is pinned by hash, so it cannot change, but if that repository is ever deleted or made private the URL returns a 404 and the Nix dev environment becomes unbuildable for all contributors.

For a file checked into the project repository, it would be worth either vendoring the relevant logic inline, using a standard pkgs.mkShell-based approach, or transferring the dependency to the online-go organization so it remains under the project's control.

@pdg137

pdg137 commented Apr 12, 2026

Copy link
Copy Markdown
Contributor Author

I think the added complexity of path traversal for something that's only used in one place isn't worth it. We can fix it later if it becomes a problem.

As for my little mkBuildableShell utility, I'm not planning to delete it and thought we'd prefer having just one nix file in the OGS project. If I ever do delete it, we could switch to mkShell or find a fork/backup of the (simple) code somewhere.

@github-actions

Copy link
Copy Markdown

rm ./node_modules fails on fresh clone

In shellHook, the else branch runs whenever node_modules does not exist (fresh clone) or when it is already a symlink. With set -eu active, rm ./node_modules will fail immediately on a fresh clone with "No such file or directory", aborting the hook before the symlink is created:

else
  echo 'Linking node_modules to Nix store...'
  rm ./node_modules          # ← fails if node_modules doesn't exist
  ln -sf ${node-modules}/node_modules ./node_modules
fi

Change to rm -f ./node_modules to handle both the absent and symlink cases safely.

@github-actions

Copy link
Copy Markdown

Review

Overall the approach is well-structured — pinned hashes for nixpkgs and the offline cache ensure reproducibility, and the vite.config.ts change is correct (the fallback node_modules/.vite matches Vite's actual default, so non-Nix developers are unaffected).

Two issues worth addressing:

1. Unquoted $@ in napi-postinstall-alias

The shell script uses $@ without quotes, which causes word-splitting on arguments containing spaces or special characters. It should be "$@":

Before:
exec ${node}/bin/node ../napi-postinstall/lib/cli.js $@

After:
exec ${node}/bin/node ../napi-postinstall/lib/cli.js "$@"

2. External dependency on a personal, non-nixpkgs repository

mkBuildableShell is fetched from pdg137/mkBuildableShell (the PR author's own repo, not in nixpkgs). The commit is pinned with a sha256 hash so existing setups are reproducible, but if that repository is ever deleted or made private, new checkouts will fail to build the Nix environment.

Options to mitigate: copy the implementation inline into default.nix, vendor the file into the repo, or document explicitly that this is a personally maintained dependency with a pointer to alternatives.

@pdg137 pdg137 force-pushed the nix branch 2 times, most recently from c9b0b94 to 6ed3f05 Compare April 12, 2026 05:13
@github-actions

Copy link
Copy Markdown

vite.config.ts — cache location change affects all developers

With root: "src" already set, Vite resolves the default cacheDir relative to that root, so the implicit pre-PR cache path was src/node_modules/.vite. The explicit path.resolve(__dirname, "node_modules/.vite") moves it to <project-root>/node_modules/.vite for everyone, not just Nix users.

That's actually the more correct location (the project's node_modules live at the root, not inside src/), but it means all developers will get a one-time Vite pre-bundling step on their next startup after pulling this change. Worth calling out in the PR description so it doesn't look like a mystery slow start.

No other issues found in the current version — the previous rm -f, "$@" quoting, and ln -sf fixes all look good.

@pdg137

pdg137 commented Apr 12, 2026

Copy link
Copy Markdown
Contributor Author

I think the latest comment by Claude is wrong. That's what it says in the docs but not how it actually works out. My change should not be a change for non-nix users. Someone else should try and confirm.

PS not knowing really anything about his, I'd support putting .vite at the project level for everyone. Shouldn't node_modules be reserved for storing node modules, not random data? But I wanted this PR to be a minimal change.

@anoek

anoek commented Apr 12, 2026

Copy link
Copy Markdown
Member

Interesting, can you update the CONTRIBUTING.md to layout how this is used? How does this compare to something like docker in terms of spinning up a quick environment on linux/mac/windows ?

@github-actions

Copy link
Copy Markdown

Thanks for contributing this! Two issues worth addressing before merge:

1. Dependency on a personal GitHub repository (default.nix)

mkBuildableShell is fetched directly from pdg137/mkBuildableShell — your personal GitHub repo. The comment in the file already flags this as a known risk. If that repo is deleted, archived, or its history is rewritten, nix-shell becomes permanently broken for anyone who tries to use it in the future, with no obvious fix available.

A safer option would be to either inline the utility (since it's described as "simple"), or fork it under the online-go organization so the project controls its own dependency.


2. vite.config.ts cacheDir change affects all developers, not just Nix users

cacheDir: path.resolve(__dirname, process.env.OGS_LOCAL_VITE_CACHE ? ".vite" : "node_modules/.vite"),

When OGS_LOCAL_VITE_CACHE is unset, this resolves to <repo-root>/node_modules/.vite. But with root: "src" set above, Vite's previous implicit default was <repo-root>/src/node_modules/.vite. Every existing developer — whether they use Nix or not — will hit a cold cache on their first yarn dev after this change.

To avoid the unintended side effect, only override cacheDir when the Nix env var is actually set:

...(process.env.OGS_LOCAL_VITE_CACHE ? { cacheDir: path.resolve(__dirname, ".vite") } : {}),

@pdg137

pdg137 commented Apr 12, 2026

Copy link
Copy Markdown
Contributor Author

Okay, I added a bit of description.

Nix has a similar goal to Docker - specifying exactly how to build and install everything, including each of its dependencies. With Nix however you are not in any kind of virtualized environment, it's just accomplished using normal environment variables like PATH and by installing each dependency in a unique location. So you can use Nix applications on your system like normal programs. You can even get NixOS, which applies this method to the entire OS.

I think Yarn achieves something similar, in a more limited scope, using the hashes in yarn.lock to lock down exact versions of all the JavaScript packages. But Nix is general and it also goes all the way down. The instructions for building the specific version of NodeJS are referenced in there, as well as the exact compiler versions, Bash, etc., with no requirement for a binary blob at any point.

@anoek

anoek commented Apr 12, 2026

Copy link
Copy Markdown
Member

I'm familiar with Nix as a full install or for a base image for vms or docker instances, but I haven't seen used for a development environment per-say.

What's the proposed flow here? Say a contributor comes along, they're running Mac OsX, what do they do to get up and running quickly?

@pdg137

pdg137 commented Apr 12, 2026

Copy link
Copy Markdown
Contributor Author

If it's someone who is already familiar with Nix and has it installed, it should be trivially easy: just nix-shell should automatically do all the work and pop them into a working dev shell.

However that's still a very niche case and for a typical person, say on macOS, I'm hesitant to suggest it since maybe your normal CONTRIBUTING instructions are already good enough. But if that has been a pain point, and assuming the Nix setup on this branch does actually work on macOS, this might offer an easier path:

  1. install Nix
  2. run nix-shell

By the way, as an example of how it can be useful, last time I was running things in this code base, we used npm run dev to start the server. I didn't know to switch yarn run dev until I read through CONTRIBUTING again. Nix provides a way to get those instructions recorded programmatically so you aren't counting on people to read. That's kind of an easy example, but if it starts to get more complicated, like if you require specific versions of things to be installed in a certain way, you can put all of that info into default.nix.

Speaking of more complicated things, I started looking at the e2e-tests and ran into this:

Error: 
╔═══════════════════════════════════════════════════╗
║ No chromium-based browser found on the system.    ║
║ Please run the following command to download one: ║
║                                                   ║
║     yarn playwright install chromium              ║
║                                                   ║
║ <3 Playwright Team                                ║
╚═══════════════════════════════════════════════════╝

Looks like that's another thing to add to the Nix config if we can figure out how to make it happen ahead of time. It's a bit troublesome that it seems to want to do a network download - is there a way to just point it at a pre-installed system chromium?

@anoek

anoek commented Apr 13, 2026

Copy link
Copy Markdown
Member

Ok, so this is strictly for those running Nix. My main concern is maintaining it in this case. Neither I nor any regular contributors use Nix so I worry this is going to rot over time and no one will notice until someone comes along and spends a few hours banging their head against outdated instructions.

So I think to get this merged we need to address that, I'm not sure if we can do that with a github action to add a build test or not, but if we can that would be great. Further we need some way of updating that default.nix to track things as they change, that's not manually editing and updating stuff - I see a bunch of sha hashes for versions and whatnot, so if we get updating that automated and some way of testing this so it's regularly exercised I think we're good to go.

@anoek anoek marked this pull request as draft April 13, 2026 13:20
@pdg137

pdg137 commented Apr 13, 2026

Copy link
Copy Markdown
Contributor Author

For automated testing, you could run e.g.

nix-shell --command 'yarn test'

which will break if someone updates yarn.lock without updating the hash correspondingly, or if the project starts to require a newer version of node/yarn than what's included presently (looks like I got v24.14.0/1.22.22).

But it's fine to just leave this alone until there's more interest. I can keep using and improving it no matter what, and it's easy for others to find in the search. Maybe one of the regulars will get curious eventually, try it out, and see the benefits.

@anoek

anoek commented Apr 13, 2026

Copy link
Copy Markdown
Member

When I say automated test I mean hooking it up into the CI, so github actions in our case. That might be very possible as you can specify base images and whatnot so it might just be a matter of selecting a nix base image and then running your above command.

@github-actions

Copy link
Copy Markdown

This pull request has been marked stale and will be closed soon without further activity. Please update the PR or respond to this comment if you're still interested in working on this.

@github-actions github-actions Bot added the stale Issues or PRs that have been open for a long time with no activity label May 14, 2026
@pdg137

pdg137 commented May 17, 2026

Copy link
Copy Markdown
Contributor Author

Updated it for the latest yarn.lock and made the update process a little smoother.

@github-actions

Copy link
Copy Markdown

vite.config.ts — cache relocation affects all developers

With root: \"src\" set, Vite's implicit default cache path is src/node_modules/.vite. Since src/node_modules/ does not exist in the repo, Vite creates that directory on first use. The PR's fallback path:

cacheDir: path.resolve(__dirname, process.env.OGS_LOCAL_VITE_CACHE ? ".vite" : "node_modules/.vite"),

resolves to <project-root>/node_modules/.vite — a different location. The inline comment says "use the default in node_modules" but this is not Vite's default; it is an explicit relocation. Every developer will lose their existing pre-bundle cache on first dev-server start after pulling this change.

The relocation is arguably more sensible (the project's node_modules is at the root, not in src/), but the side effect should be noted in the PR description and the comment corrected.

If the goal is to minimize impact on non-Nix users, only override cacheDir when the env var is set:

...(process.env.OGS_LOCAL_VITE_CACHE ? { cacheDir: path.resolve(__dirname, ".vite") } : {}),

@github-actions

Copy link
Copy Markdown

Nix string interpolation bug in default.nix

Line 69 uses Ruby-style interpolation syntax instead of Nix syntax:

The name field reads: online-go-offline-cache-#{newYarnLockHash}

In Nix, string interpolation uses dollar-brace syntax, not hash-brace. The hash is never substituted, so the derivation name is the literal string containing the characters hash-open-brace-newYarnLockHash-close-brace. The newYarnLockHash binding computed on the line above is therefore unused dead code.

Correct Nix syntax: name = "online-go-offline-cache-${newYarnLockHash}";

@github-actions

Copy link
Copy Markdown

default.nix — inconsistent Node.js version in build derivation

The node variable is defined as pkgs.nodejs_24, used consistently in napi-postinstall-alias and the shell's buildInputs. However, the node-modules build derivation's nativeBuildInputs lists pkgs.nodejs instead of node:

nativeBuildInputs = [
  pkgs.nodejs        # default — likely Node 22 in nixpkgs-25.11, not Node 24
  pkgs.yarn
  pkgs.fixup-yarn-lock
  napi-postinstall-alias
];

This puts a different node binary on PATH during the build than the one the shell provides at runtime. Any package install script that calls node from PATH (other than unrs-resolver, which is handled by the explicit napi-postinstall-alias) will run under the wrong version. If such a script selects or compiles a native binary based on the Node version, the resulting artifact will be incompatible at runtime.

Fix: replace pkgs.nodejs with node (i.e. pkgs.nodejs_24) in nativeBuildInputs.

@github-actions github-actions Bot removed the stale Issues or PRs that have been open for a long time with no activity label May 18, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been marked stale and will be closed soon without further activity. Please update the PR or respond to this comment if you're still interested in working on this.

@github-actions github-actions Bot added the stale Issues or PRs that have been open for a long time with no activity label Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale Issues or PRs that have been open for a long time with no activity

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants