OPERATIONS.md

Operations

Setup

System Dependencies

The hooks require these tools (ship with most systems):

git jq python3 sed grep

Python Dependencies

python -m pip install --require-hashes -r requirements.lock

Core installs are hash-pinned via requirements.lock. Update the lock whenever requirements.txt changes:

uv pip compile --generate-hashes requirements.txt -o requirements.lock

Core: PyYAML==6.0.3; test tooling: pytest (runs the tests/ regression suite). Optional per-skill extras are documented in requirements.txt.

Run the repo regression tests with:

python -m pytest tests/ -q

Skill Management Commands

Create a new skill

python skills/skill-creator/scripts/init_skill.py <skill-name> --path ./ \
  --resources scripts,references --examples

Validate a skill

python skills/skill-creator/scripts/quick_validate.py <path/to/skill-folder>

Validate a spec or plan

The pre-execution pipeline is brainstorm (docs/design/) → spec (docs/specs/) → plan (docs/plans/). Specs are mechanism-free contracts; plans are the execution breakdown. Each layer has its own schema validator (also wired as on-write hooks):

# Contract schema — rejects plan-shaped content (files/steps/task breakdowns)
python3 skills/write-spec/scripts/validate_spec.py docs/specs/<file>-spec.md

# Execution schema — task breakdown, files, ordered steps, verification matrix
python3 skills/write-plan/scripts/validate_plan.py docs/plans/<file>-plan.md

Add --strict-filename to enforce the -spec.md / -plan.md suffix (the hooks do). Both validators print non-blocking advisories for obvious weak acceptance language; the plan validator also advises on grounding and test discovery. These messages never change an otherwise valid artifact's exit status. Plan spec: and Modify: paths resolve from the target plan's Git root; pass --repo-root <path> for relocated artifacts, or invoke from the target root when working outside Git.

New artifacts declare risk_profile: routine|high and readiness: draft|ready; legacy artifacts default to routine/draft. High-risk specs add stable criteria/scenario IDs and critique closure. High-risk plans link the spec through repository-relative spec: frontmatter and hard-fail on missing ID coverage, task dependencies, modified files, conditional readiness sections, or unresolved blocking findings. Semantic authority and recovery claims still require adversarial review.

Every newly authored design summary, spec, or plan also declares author: with the producing agent's most specific available model or harness identifier (for example, gpt-5.6-sol); the literal <agent> template placeholder must be resolved. Spec/plan validators enforce this for current-schema artifacts while continuing to accept legacy documents without fabricated attribution.

Package a skill for distribution

python skills/skill-creator/scripts/package_skill.py <path/to/skill-folder> [output-directory]

Generate OpenAI metadata (optional)

python skills/skill-creator/scripts/generate_openai_yaml.py <path/to/skill-folder>

Regenerate manifest

python scripts/generate_skills_manifest.py          # write manifest + refresh catalog
python scripts/generate_skills_manifest.py --check  # verify manifest drift (CI)

The top-level skills.json version is the manifest schema version. Each skill entry also includes the per-skill release version declared in SKILL.md frontmatter.

The write path also refreshes the browseable catalog (docs/catalog/index.html), which is derived entirely from skills.json, so the two never drift after a manual regen. Pass --no-catalog to skip it, or --catalog <path> to target a different file; gen_catalog.py --check remains the catalog's own CI drift gate.

Check skill release versions

Compares release-relevant skill changes against a git base. After the one-time unversioned baseline, changed skill contents require a strictly greater SemVer value and a matching CHANGELOG.md entry:

python3 skills/skill-evals/scripts/check_skill_versions.py --base origin/main

Use DOJO_VERSION_CHECK_BASE=<ref> to point the stop hook at a different comparison base when working from a non-main integration branch.

Not release-relevant (changing these alone needs no bump): CHANGELOG.md, the generated agents/openai.yaml sidecar, __pycache__/.pytest_cache and .pyc/.pyo artifacts, and the append-only run-memory files references/postmortems.md and references/executor-profiles.md — a skill accumulates those as it is used, not as it is changed.

Bump a skill version

Performs the two-file edit the version check requires — updates the version field in SKILL.md frontmatter and prepends a matching ## <version> - <date> heading (and bullet) to CHANGELOG.md, creating it if absent — and then, on a successful non-dry-run bump, regenerates skills.json and refreshes docs/catalog/index.html so a bump never leaves generated artifacts stale for CI's --check steps:

python3 skills/skill-evals/scripts/bump_skill_version.py skills/<name> patch   # or minor / major
python3 skills/skill-evals/scripts/bump_skill_version.py skills/<name> minor -m "What changed."
python3 skills/skill-evals/scripts/bump_skill_version.py skills/<name> --set 2.0.0   # explicit version
python3 skills/skill-evals/scripts/bump_skill_version.py skills/<name> patch --dry-run   # no writes
python3 skills/skill-evals/scripts/bump_skill_version.py skills/<name> patch --no-regen  # skip regen

Because the bump writes SKILL.md directly (the post-tool-use manifest hook does not see it), the regeneration is built in — so a normal bump touches up to four tracked files (SKILL.md, CHANGELOG.md, skills.json, docs/catalog/index.html). Pass --no-regen for batch bumps and run scripts/generate_skills_manifest.py once at the end instead of regenerating per bump. Regeneration no-ops with a printed reminder when the skill is bumped outside a dojo checkout (e.g. a global install, where the generator is absent).

Bumping resets lower parts and drops any prerelease (1.2.0-rc.1 patch → 1.2.1). It refuses a non-increasing --set and refuses to duplicate an existing changelog heading. -m/--message supplies the entry bullet (a placeholder is written otherwise).

Compose opt-in shared fragments

Expands <!-- INCLUDE: name --> directives from skills/_fragments/ into SKILL.md (skills with no directive are untouched):

python scripts/gen_skill_docs.py          # write
python scripts/gen_skill_docs.py --check  # verify no drift (CI)

Recover an earlier version of a skill

Use git. Every canonical version of every skill is in this repository's history, and that is the answer in almost all cases:

git log --oneline -- skills/<name>              # what changed and when
git show <sha>:skills/<name>/SKILL.md           # read that version
git restore --source=<sha> skills/<name>        # bring it back

sync.py --apply also writes a timestamped backup of whatever it overwrote, under --backup-root (default .skill-standardizer/backups/, gitignored). Those hold the one thing git cannot: an installed copy that had drifted from canonical. That is worth little once drift is being watched, so they are pruned to the most recent --keep-backups runs (default 10; 0 keeps everything).

Other backup directories on these machines are not managed by this repository and are left alone: ~/.agents/.skill-backups/ and ~/.agents/.backups/ are ad-hoc, and ~/.claude/backups/ belongs to Claude Code itself (its own .claude.json snapshots, nothing to do with skills).

Regenerate harness adapters

Creates the local .claude/.agent skills/ symlinks and the colocated Codex openai.yaml sidecars from frontmatter. Also retires a legacy .agents/skills catalog link if one survives from an older checkout (a real directory or a foreign symlink is reported, never deleted). Run after cloning (symlinks are gitignored) and after editing skill descriptions:

python scripts/gen_harness_adapters.py                      # write symlinks, .claude/commands links, and sidecars
python scripts/gen_harness_adapters.py --check              # verify all locally
python scripts/gen_harness_adapters.py --check --skip-symlinks  # verify committed sidecars only (CI)

Hand-curated sidecars (no AUTO-GENERATED marker) are preserved; for those, author with skills/skill-creator/scripts/generate_openai_yaml.py.

The same generator links each skill's commands/*.md into .claude/commands/ (local-only, gitignored) so Claude Code exposes them as slash commands. It refuses when two skills' commands map to the same name (rename one), prunes symlinks whose source was removed, and never touches a hand-authored file in .claude/commands/. Commands are governed by the symlink phase, so --skip-symlinks (CI) ignores them.

Regenerate the skill catalog

Builds a self-contained, searchable docs/catalog/index.html from skills.json (open it directly in a browser):

python scripts/gen_catalog.py          # write
python scripts/gen_catalog.py --check  # verify no drift (CI)

Scan for AI-slop prose

Deterministic, high-precision linter for AI-slop tells in skill prose + core docs (complements the visual design-critique skill):

python scripts/slop_scan.py          # scan default set; exit 1 on hits (CI)
python scripts/slop_scan.py --list   # show the patterns
python scripts/slop_scan.py PATH...  # scan specific files

Skill health report

Read-only aggregation of contract status + declared-trigger routing across the catalog (reporting, not a gate):

python scripts/skills_health.py         # human-readable
python scripts/skills_health.py --json  # machine-readable

The default run is network-free and unchanged. Opt-in runtime flags enrich the report with per-skill trigger health from a running AgentMonitor instance (the sibling local observability console) via GET /api/v2/analytics/skills/health — invocation counts, never-fired status, and an experimental misfire rate:

python scripts/skills_health.py --runtime                 # fetch localhost:3141 (default endpoint)
python scripts/skills_health.py --agentmonitor-url URL    # custom endpoint (implies --runtime)
python scripts/skills_health.py --health-json FILE        # offline: read a saved health payload
python scripts/skills_health.py --findings --runtime      # paste-ready BACKLOG blocks for never-fired skills

Any runtime flag activates the runtime path; the default endpoint is http://127.0.0.1:3141/api/v2/analytics/skills/health. The runtime section ranks by the trustworthy signals — never-fired first, then a rarely-fired band, then invocation volume ascending — while misfire is shown labeled experimental and never drives rank. If AgentMonitor is unreachable or returns an unexpected shape, the tool exits non-zero with a diagnostic and prints no partial report. --findings only proposes maintainer-reviewable blocks; it writes nothing and never invokes skill-evals.

Behavioral trigger evals (opt-in, never in CI)

Asks a real local agent which skill it would pick for each declared trigger, then checks against the owner. Requires DOJO_BEHAVIORAL_EVALS=1 and a local agent command (DOJO_BEHAVIORAL_AGENT, default claude -p; reads the prompt on stdin). Non-deterministic and may cost tokens, so it is gated off by default and not wired into CI:

DOJO_BEHAVIORAL_EVALS=1 python scripts/behavioral_evals.py
DOJO_BEHAVIORAL_EVALS=1 python scripts/behavioral_evals.py --json

Run skill-standardizer regression tests

This suite ships beside the skill rather than in tests/, so pytest tests/ does not collect it. CI runs it as its own step; invoke it directly to run it locally:

python3 skills/skill-standardizer/scripts/test_skill_standardizer.py

Measure what a session's skill listing actually costs

Both harnesses expose their model-visible listing deterministically, with no LLM in the loop. This answers a question nothing else in the repo answers — what the effective catalog costs right now, on this harness, in that harness's own arithmetic.

# Codex: entries, origins, and utilisation against the 2%-of-window budget
.venv/bin/python scripts/profiles/probe_codex.py --cwd . [--json]

# Claude Code: `sent` is the listing count, `demand_chars` the untruncated figure
.venv/bin/python scripts/profiles/probe_claude.py --cwd . --model opus [--json]

Four things to know before trusting a number from these:

• The Codex probe reads the exec surface, not the interactive one — do not quote it as a session cost until Task 5A lands. codex debug prompt-input does not load account-synced connector plugins that codex-tui loads. On 2026-08-04 the same directory, model, and minute measured 41 entries at 76% via the probe and 110 entries at 246% with every description clipped to ≤77 characters in the actual TUI session. The budget is also wrong from the probe: interactive renders on build 0.146.0 saturate at exactly 4,000 tokens, not the 5,440 that codex debug models' 272,000 window implies. 0.147.0 raised it again — a 4,843-token render no longer saturates, so the limit there is only known to be >= 4,843 until something clips. The ceiling is a property of the CLI build — 0.143.0/0.144.x saturate at 5,440 (and at 7,440 for a 372,000-window model, both matching 2% × window), and 0.145.0 changed it. Never pool samples across builds. And the harness's shortening warning is not a fit signal — Codex warned at 246% and stayed silent at 144% while still clipping 50 of 56 descriptions. Until the observation moves to the rollout record, read a live session instead:

  # what the harness ACTUALLY sent, per session, with its originator
  ls -t ~/.codex/sessions/*/*/*/rollout-*.jsonl | head -1

• demand is computed from untruncated source, never from the rendered listing. A harness that elides to fit produces output that always fits: a live capture rendered 8,046 characters against an 8,000 budget while true demand was 23,287. Anything calibrated on rendered output reports ~100% for a 291% listing.

• Claude Code's budget scales with the model's context window — 8,000 characters at 200k, 40,000 at 1M — so a figure is meaningless without the model. Pass --model deliberately.

• The filesystem is not the listing — and neither is a different code path. A skill can be installed and unlisted (microsoft-foundry, disabled) or bundled and unlisted (review-agent); neither costs anything. But the converse now has a live case too: openai-templates holds 20 skills worth 1,870 tokens (34% of budget) that appear in none of eleven captured sessions and are indistinguishable on disk from connectors that do list. Account-synced connectors are invisible to codex plugin list entirely, and are governed by a <plugin>@openai-curated-remote config key rather than the <plugin>@openai-curated key that CLI displays — so "disabled" there can be true and irrelevant at the same time.

The profile library under scripts/profiles/ composes these into resolution, budget assessment, observation, and a byte-identical evidence report. It is read-only throughout — no command in it changes installed state. The individual entrypoints are the interface; the unified dojo profiles verify --all wrapper (Task 8) was dropped when the distribution-profiles contract closed at shipped scope (spec revision 16) — it would only have wrapped a measurement the audit repeatedly proved could be wrong, so it solved no problem the library does not already expose.

CI gate — runs on every pull request touching profiles/**, scripts/**, skills/**, tests/**, or the docs trees:

python3 scripts/profiles/ci_check.py

It validates definitions, resolves every declared composition, loads each budget policy, refuses a policy declared deployable on a limit never checked against behaviour, and scores core+engineering so a green run cannot mean it evaluated nothing. It then prints what it did not check. That boundary is the point: CI has no harness binary and no session rollouts, so it cannot observe an effective catalog at all. Every failure this program actually hit — a ceiling that moved three times in eight days, a locator convention that reversed between builds, an account connector sync that refilled recovered headroom overnight — is invisible from CI by construction.

Machine-side drift check — those failures need a machine with a harness on it, so they are watched there instead:

python3 scripts/profiles/drift_check.py            # report against the baseline
python3 scripts/profiles/drift_check.py --update   # accept the current state
python3 scripts/profiles/drift_check.py --json     # machine-readable
python3 scripts/profiles/drift_check.py --uncontrolled-as-notice  # health mode

It compares the newest codex-tui rollout against a recorded baseline (default ~/.agents/.dojo-profile-baseline.json, deliberately outside this public repository, and keyed by working directory) and reports build changes, model changes, a moved or newly underivable ceiling, saturation starting or stopping, changes to the listed and uncontrolled entry multisets, and charged demand moving under unchanged membership. Exit 0 clean, 1 cannot evaluate, 2 drift, 3 blind too long, 4 saturated.

The default remains strict. --uncontrolled-as-notice exits 0 with state: notice only when the same-build, non-saturated transition is entirely vendor/account membership churn and the controlled catalog, surface, ceiling, and saturation state are unchanged. Exact findings remain in the report; build, controlled membership, ceiling, and saturation changes remain failures. This mode is intended for machine-health wrappers, where packaging churn is observable without declaring the workstation unhealthy.

It fails closed: no rollout, an unparseable one, a missing baseline, or degraded origin classification all report cannot evaluate, never clean — a monitor that reports "no drift" when it observed nothing is worse than no monitor. --max-blind-days N extends that to the wrapper's problem: a machine between interactive sessions has nothing to compare and stays quiet, while one that has never been observed, or not in N days, exits 3. Without that distinction a scheduled check reports healthy forever on a machine nobody uses interactively — which is exactly what the mini did for its first week.

Saturation (4) is a state, not a change. Exit 2 answers "did it move?"; a listing that was already clipping when the baseline was recorded has not moved, so on 2026-08-13 the mini reported state: clean while cutting 24 of 48 descriptions mid-word. Exit 4 answers "is it broken?", is derived from the current observation alone, and repeats on every run until the catalog fits or the build's ceiling rises — --update cannot accept it into a baseline the way it accepts drift. Precedence, when more than one holds:

	outranks	because
3 blind	4 saturated	saturation is a present-tense claim, and a stale sample is not evidence of the present
4 saturated	2 drift	--update accepts drift, so reporting clipping as drift announces it once and then debounces it into silence
2 drift	a stale 3	"A differs from B" stays true however old A and B are

The report always carries every finding; precedence only decides which word the exit code says.

It runs in two places, because neither alone is enough:

• SessionStart hook (hooks/session-start-harness-drift.sh) on whichever machine you are working on. The data it reads is written by interactive sessions, so the daily driver is the only place it reliably sees anything. It is silent on clean and on cannot-evaluate, and speaks on drift, persistent blindness, or saturation. --update gives it debounce for free: a change is reported once and then accepted, so a Codex upgrade does not nag every session. Saturation is deliberately exempt from that debounce — the clipped listing it describes is the one the session is about to run with.
• Scheduled from a deployment's recurring machine-health wrapper, which can tell "quiet this week" from "never" and so carries --max-blind-days. Such a wrapper can also opt into --uncontrolled-as-notice; the interactive hook stays strict.

Baselines are stored per working directory because a listing is not a property of the machine alone: Codex loads a project's own .agents/skills on top of the global root. Measured at build 0.144.1, ~/Dev listed 55 entries while ~/Dev/dojo listed 112. A single shared baseline would accept whichever project was seen last and report ~57 entries added on the next switch, then the reverse coming back — the check would fire on project switching rather than on drift. This is the rule already applied to builds, extended to the other axis a sample belongs to: never compare observations that were never comparable. A baseline file written before this keying is dropped and re-seeded rather than guessed at, which costs one silent cycle.

It fails closed: no rollout, an unparseable one, a missing baseline, or degraded origin classification all report cannot evaluate, never clean — a monitor that reports "no drift" when it observed nothing is worse than no monitor. The one exception runs the same way: whether the current listing clips is knowable from that listing alone, so a missing baseline reports saturation (4) rather than cannot evaluate when it does. Falling back to 1 there would be the same silence by another door, because the scheduled wrapper treats it as a pass.

A recurring health wrapper should run it with --update, so a change is reported once and then accepted as the new normal; without that, one Codex upgrade would re-report the same drift every cycle until someone silenced the job. Notice mode changes only the severity of uncontrolled packaging churn, not the saved evidence. Saturation is deliberately outside that debounce — --update settles what to compare against next time, and says nothing about whether the sample was healthy.

Hook Configuration

Hooks are configured in .claude/settings.json and .agents/settings.json. No manual installation is needed — they activate automatically when opening the repo in a supported harness.

The SessionStart drift notice (session-start-skill-drift.sh) keeps per-checkout debounce state in .skill-standardizer/drift-state.json (gitignored). To force the next session to re-report current drift, delete that file.

CI

GitHub Actions enforces strict contract compliance, generated-artifact sync, and an AI-slop prose scan on the manifest-backed skill catalog via:

• .github/workflows/skill-contract-pilot.yml

python -m pytest tests/ -q
python3 skills/skill-standardizer/scripts/test_skill_standardizer.py
python3 skills/skill-evals/scripts/validate_skill_contract.py --skills-root skills --strict
python3 skills/skill-evals/scripts/check_skill_versions.py --base origin/main
python3 scripts/gen_skill_docs.py --check
python3 scripts/generate_skills_manifest.py --check
python3 scripts/gen_harness_adapters.py --check --skip-symlinks
python3 scripts/gen_catalog.py --check
python3 scripts/slop_scan.py

The strict validator is type-aware:

• workflow skills must define execution flow and output expectations.
• reference skills are evaluated on scope, boundaries, verification, and resource navigation without being forced into workflow-only sections.

Hooks enforce quality at edit-time and session-stop:

• Pre-tool-use hook blocks pushes to protected branches unless the command includes DOJO_ALLOW_PROTECTED_PUSH=1.
• Pre-tool-use hook validates SKILL.md on every write.
• Stop hooks verify git state, skill structure, and required skill version bumps; uncommitted changes and untracked files block, but unpushed commits never do (push timing is the operator's call).

See docs/system/skill-contract-v1.md for the contract checklist and docs/system/SKILL-BEST-PRACTICES.md for authoring guidance.

Optional Skill Dependencies

Skill	Extra Packages	Env Vars
gemini-imagen	google-genai>=1.0.0, Pillow>=10.0.0	GEMINI_API_KEY
gpt-imagen	openai>=1.0.0, Pillow>=10.0.0	OPENAI_API_KEY
design-md	npx on PATH; pulls @google/design.md@0.1.1 on first invocation via scripts/run_cli.sh	—