skillsmith

Tune a coding agent's skill against your repo's real tests — keep only the edits that measurably raise the pass rate, graded by executing the tests, not by an LLM judging text.

No API key needed — the default provider uses your installed claude CLI's own auth.

Overview

A coding agent's skill is a markdown instruction sheet — conventions, gotchas, how-tos — that you hand the model. skillsmith treats that document as the artifact to optimize and refuses any edit that doesn't measurably help:

baseline → propose skill edits → re-eval (run tests) → gate (keep only if pass rate ↑) → stage

Every candidate is graded by running the repo's own tests (verify_cmd) inside an isolated git worktree; the score is the fraction of test cases that pass (a continuous signal, not just pass/fail), so the optimizer climbs a real gradient. Nothing live changes — the improved skill is staged until you adopt it.

• Measured on your code, not a benchmark. The reward is your verify_cmd — no gold-answer strings, no LLM grading prose it can game.
• Un-gameable gate. A test suite isn't swayed by confident writing; an edit is kept only if execution says it's better.
• Generalization, not memorization (opt-in). Hold out validation tasks so an accepted edit has to transfer, not just fit the cases it saw.
• Zero-config & local. cargo install, run from anywhere, the demo auto-seeds, no API key; every candidate runs in a throwaway git worktree.

How it works

flowchart LR
  CFG([project<br/>tasks + seed skill]) --> BASE[baseline eval]
  BASE --> RND{{round 1..N}}
  RND --> PROP[optimizer proposes<br/>bounded skill edits]
  PROP --> CAND[candidate skill]
  CAND --> EVAL
  subgraph EVAL [eval candidate · per task]
    direction TB
    AG[agent edits code<br/>using the skill] --> WT[apply in isolated<br/>git worktree]
    WT --> VC[run verify_cmd<br/>score = cases passed]
  end
  EVAL --> GATE{gate score ↑<br/>vs best?}
  GATE -- yes --> KEEP[accept · new best]
  GATE -- no --> REJ[reject · keep best]
  KEEP --> RND
  REJ --> RND
  RND -- rounds done --> STAGE[stage skill.staged.md<br/>+ report.md]
  STAGE --> ADOPT([you adopt:<br/>skillsmith adopt])

Loading

The gate is strict — a candidate is kept only if it raises the measured score over the current best. The execution judge (agent edits applied in a throwaway worktree, then verify_cmd run and scored) is what makes the lift real rather than a model's opinion of its own output.

Quickstart

Engine first, the skill just calls it.

# prereqs: Rust + git + an LLM backend — an agent CLI (claude/codex/gemini, no key)
#          OR ANTHROPIC_API_KEY for the genai provider; python3 for the demo
git clone <repo-url> skillsmith && cd skillsmith
cargo install --path .          # puts `skillsmith` on PATH (~/.cargo/bin); no env var

skillsmith run --project demo --dry-run   # preflight: no LLM, no tokens
skillsmith run --project demo             # demo auto-seeds into ~/.skillsmith on first run

The bundled demo proves the loop: slugify passes at baseline; house_join fails (its separator " :: " is withheld from the prompt and the held-out test), so the optimizer learns it from the failure → candidate passes → measurable lift, gate accepts. The improved skill is staged to ~/.skillsmith/projects/demo/skill.staged.md; nothing live changes until you skillsmith adopt --project demo (copies the staged proposal over the project's skill.md).

The first real run needs an LLM backend (--dry-run, list, new don't): either log into the claude CLI (default provider = "claude", no key; codex / gemini likewise), or set provider = "genai" + export ANTHROPIC_API_KEY (best for CI / headless). Backends, models, and env vars: Providers.

Home resolves as --home > $SKILLSMITH_HOME > a .skillsmith/ found above your cwd (repo-local) > ~/.skillsmith. The demo auto-seeds only into the per-user default (never a repo-local .skillsmith/); skillsmith init reseeds it.

Walkthrough (end-to-end)

The whole flow in order, for a newcomer optimizing their own repo (each step links to its reference section below). Only run spends tokens — dry-run / adopt / deploy / list / new don't.

1. Install the engine (Quickstart):

   git clone https://github.com/JHoo0118/skillsmith && cd skillsmith
   cargo install --path .          # skillsmith -> ~/.cargo/bin (needs ~/.cargo/bin on PATH)

2. Register /skillsmith (Run from your agent CLI) — the marketplace plugin or just install (the only route for Gemini).
3. Set a backend for the first real run: log into the claude CLI or export ANTHROPIC_API_KEY (Quickstart callout above). dry-run needs neither.
4. Sanity-check on the demo — proves the loop, no risk to your repo:

   skillsmith run --project demo --dry-run    # 0 tokens
   skillsmith run --project demo              # ~5 calls; house_join lift

5. Author a project for your repo (interactive) — don't hand-write config; tell the agent and confirm the target (HITL):

   /skillsmith optimize this repo with skillsmith
   

   Green tree? The agent builds a fixture (stub + held-out test) so the baseline can fail — that gap is the headroom.
6. Run the optimization:

   /skillsmith run --project <name>           # baseline -> best (lift); stages skill.staged.md

7. Adopt the staged skill — HITL, the one live-changing step:

   /skillsmith adopt --project <name>         # skill.staged.md -> the project's skill.md

8. Deploy it where an agent reads it (After adopt → deploy) — HITL, no tokens:

   /skillsmith deploy --project <name> --desc "<trigger phrases>"   # -> .claude/skills/<name>/SKILL.md

   A new session auto-loads it when the description matches.

Copy-paste, on your own repo:

cd /path/to/repo
/skillsmith optimize this repo with skillsmith   # author + HITL + dry-run (the agent does it)
/skillsmith run    --project <name>              # measure (tokens)
/skillsmith adopt  --project <name>              # accept (after HITL)
/skillsmith deploy --project <name> --desc "…"   # publish to the agent (no tokens)

Two common snags: ~/.cargo/bin not on PATH (open a new shell), and no backend set for the first run (step 3).

Use it on your own repo

Two layouts — repo-local (recommended: a skill is project knowledge, so version it with the repo) or central (under ~/.skillsmith, nothing added to the repo).

# repo-local — a .skillsmith/ in your repo, auto-discovered like .git
cd /path/to/your/repo
skillsmith new                     # bare: repo-local + named after the repo dir
#   edit ./.skillsmith/projects/<name>/config.toml  (eval tasks; repo_path = this repo)
#   edit ./.skillsmith/projects/<name>/skill.md      (seed conventions — commit it)
skillsmith run --project <name> --dry-run
skillsmith run --project <name>

# central — keep every project under ~/.skillsmith, point at the repo by path
skillsmith new myproj --repo /path/to/your/git/repo
skillsmith run --project myproj

skill.md + config.toml are meant to be committed; the generated skill.staged.md / report.md / results.json are gitignored for you. Each task runs in an isolated git worktree; your verify_cmd decides pass/fail. Keep the test file out of context_files (held out) so the skill carries the knowledge. The target repo must have ≥1 commit.

Preflight (no tokens): --dry-run runs each verify_cmd in a worktree without the LLM — catch a broken repo_path / verify_cmd before spending agent calls.

When your repo is already green → build a fixture

If every test already passes, baseline = 1.0 → no headroom → no lift. The optimizer needs a failing start. Either bind to a genuinely unimplemented / failing function, or build a small fixture — a separate, committed git repo that stubs one function and holds the real behavior as a held-out test:

projects/<name>/
  config.toml          # repo_path = "fixture"
  skill.md             # seed — vague; the optimizer fills it in
  fixture/             # its OWN git repo (git init + commit it)
    thing.py           # def f(...): raise NotImplementedError   ← the stub
    test_thing.py      # the REAL behavior as assertions         ← held-out oracle

• context_files = ["thing.py"] → the agent sees the stub, not the answer, so it must reconstruct f from the intent + skill. That gap is the headroom.
• verify_cmd runs test_thing.py; keep that test out of context_files.
• Encode the real convention, not an invented one. A made-up prefix/separator will "lift" but won't transfer to your actual code. Use the format/constant your codebase really uses, so the optimized skill is portable (deployable — below).

This is exactly what the demo's house_join is, generalized to your own repo.

After adopt → deploy the skill

skillsmith adopt copies the staged skill over the project's skill.md and stops there — nothing auto-reads that file. To make a code agent actually use the validated rules, deploy them where the agent reads:

path	what	where	who reads it
A. Promote to context	the validated rule (short)	CLAUDE.md / AGENTS.md / GEMINI.md	all agents, always-on
B. Install as a skill	the skill, wrapped in name + description frontmatter	.claude/skills/<name>/SKILL.md (Codex/Gemini: own dirs)	on-demand when the description matches
C. Inject into a prompt	@path to the skill	a specific agent's system prompt	that one agent

Rule of thumb: short + always-applies → A (the only path all three agents load automatically); long + task-narrow + Claude-primary → B (auto-trigger is first-class on Claude; Codex/Gemini approximate it with manual prompts/commands).

skillsmith deploy does A and B for you — pure file ops, no tokens:

skillsmith deploy --project <name>                                   # B: .claude/skills/<name>/SKILL.md
skillsmith deploy --project <name> --as context --agents claude,codex,gemini   # A: inject into CLAUDE/AGENTS/GEMINI.md

Re-deploy is idempotent (the context block updates in place between markers). Give the skill real trigger phrases with --desc "..." (or a [deploy] description = "..." in config.toml) so it auto-triggers — a placeholder is written with a warning otherwise. C (prompt injection) stays manual.

Keep it fresh — drift check (no tokens)

A skill is optimized against specific files; when they change, it can drift out of date. skillsmith check --project <name> compares the repo inputs (config + target/context files — not the skill itself, which adopt rewrites) against the last run and exits non-zero on drift. It's detection only — never re-runs or adopts (that stays your call, and is the only step that spends tokens). Wire it into a non-blocking git pre-commit hook:

# .git/hooks/pre-commit (chmod +x) — warn on a stale skill, never block the commit
for cfg in .skillsmith/projects/*/config.toml; do
  name=$(basename "$(dirname "$cfg")")
  skillsmith check --project "$name" || true
done

This is the safe, frugal half of "auto-strengthen on change": detect + notify here, re-run + adopt stay your explicit call.

Run from your agent CLI

Register the /skillsmith command once; it's a thin wrapper that calls the PATH binary, so adding or editing projects needs no re-registration.

# Claude Code — marketplace plugin (point at the plugin dir; ABSOLUTE path, not a bare "."):
/plugin marketplace add /abs/path/to/skillsmith/plugins/skillsmith   # the plugin dir in the clone you built the engine from
/plugin install skillsmith@skillsmith

# Codex — marketplace plugin:
codex plugin marketplace add ./plugins/skillsmith   # finds .agents/plugins/marketplace.json there
codex /plugins                                       # → Install "skillsmith"

# Any agent — symlink install (no marketplace; required for Gemini):
just install        # both + Codex + Gemini  (or: install-claude | install-codex | install-gemini)

Agent	Invoke
Claude Code	/skillsmith run --project demo
Codex	@skillsmith run demo (or just describe the task)
Gemini CLI	/skillsmith run --project demo (after /commands reload)

Prereq: cargo install --path . once so the skillsmith binary is on PATH (the command calls it). Re-run it only when you change skillsmith's own code — that updates the engine, not the command registration. A marketplace install is cached (/plugin marketplace update to refresh the markdown); a symlink install is live.

Author a project interactively (no hand-written config). No project yet? Don't hand-write config.toml — describe the job as the command's argument and let the agent author it. Vague or fully-scoped, both work:

# vague — skillsmith scans your repo and proposes a target:
/skillsmith optimize this repo with skillsmith

# scoped — name the function, the strategy, and where to stop, in one request:
/skillsmith create a project for normalize_url in src/net/util.py; the tree is green, so use a fixture (stub + held-out test), then dry-run

These are free-form authoring requests the skill interprets — not flag subcommands (those are /skillsmith run|eval|adopt|list|new …). Either way the host agent (Claude / Codex / Gemini) can read your repo and ask you questions — while skillsmith's internal judge agent is deliberately blind — so it picks (or confirms) a small function + focused test, confirms the target with you (HITL), and generates config.toml + a seed skill.md (repo-local, the answer withheld from the intent). Then it dry-runs (no tokens) and runs. (Green tree → it builds the fixture for you — see above.) You only hand-write configs when driving the bare binary.

The command + skill live once under plugins/skillsmith/ and feed all integrations — everything plugin-related lives under plugins/: the package plus its manifests (Claude plugins/skillsmith/.claude-plugin/, Codex plugins/skillsmith/.agents/ + .codex-plugin/) and the Gemini command (plugins/gemini/). Details: plugins/README.md.

Providers (no API key by default)

Set provider in the project's config.toml:

provider	uses	auth
claude (default)	claude -p	your Claude Code login — no key
codex / gemini	codex exec / gemini -p	that CLI's login — no key
genai	raw API (genai crate)	an API key (varies by model — below)
cli	custom provider_cmd	the tool's own

genai is the only provider that needs a raw key — claude / codex / gemini reuse the CLI's own login (so codex does not read OPENAI_API_KEY). For genai, which env var depends on the model (it routes by model prefix on agent_model / optimizer_model):

model family	env var
claude-* (skillsmith's default)	ANTHROPIC_API_KEY
gpt-* / o3-* …	OPENAI_API_KEY
gemini-*	GEMINI_API_KEY

Defaults are claude-* → just export ANTHROPIC_API_KEY. To use another backend, set agent_model / optimizer_model to that family and export its key. For large or automated batch runs, genai keeps you within API rate limits rather than your subscription's.

Tier the cheap stage (optional). The agent (eval) stage makes far more calls than the optimizer (propose) stage, so running it on a smaller model saves the most — genai tiers by agent_model, CLI providers by agent_provider_cmd (see Config below).

Config (projects/<name>/config.toml)

name = "myproj"
repo_path = "/path/to/repo"   # git repo to worktree; relative -> vs the project dir;
                              #   omit (repo-local mode) -> the git repo enclosing .skillsmith/
skill_file = "skill.md"       # the artifact being optimized
provider = "claude"
rounds = 3

[[task]]
id = "example"
intent = "What the agent must do (don't leak the answer)."
context_files = []            # shown to the agent — NOT the test file
target_files  = []
# setup_cmd = "..."                   # optional: seed worktree state before the agent's edits
verify_cmd = "pytest tests/x.py -q"   # graded: fraction of test cases passing
# holdout = true                      # = split="val": gate on this (optimizer never sees it)
# split = "test"                      # held out entirely; best skill scored on it ONCE at the end (unbiased)

# [deploy]                            # optional defaults for `skillsmith deploy` (then no flags needed)
# name = "my-skill"                   # skill/block name (default: the project name)
# description = "trigger phrases…"    # frontmatter description / when an agent should auto-load it

agent_model / optimizer_model apply to the genai provider; CLI providers ignore them and tier per-stage via agent_provider_cmd / optimizer_provider_cmd instead — e.g. ["claude", "-p", "--model", "claude-haiku-4-5"] or ["codex", "exec", "-m", "gpt-5-mini"] (omit → base provider).

Scoring is graded — the score is the fraction of individual test cases that pass (parsed from pytest/unittest output; falls back to the exit code for non-test commands). Each task has a split: train (default — the optimizer learns from its failures), val (held out; the gate scores accept/reject on these — holdout = true is the alias), or test (never run during optimization; the best skill is scored on it once at the end for an unbiased number). With no val/test tasks, everything is train + gate-on-all.

Machine-readable results — each run writes results.json next to the human report.md: run metadata (provider, models, config_fingerprint, llm_calls, elapsed), baseline/best/test per-task scores, test_score (the unbiased held-out number), and per-round gate decisions. The seam for CI gates, cross-run sweeps, and benchmarking (schema_version is bumped on shape changes).

Benchmark (k-seed variance) — skillsmith bench --project <name> --seeds N runs the optimization N times (the agent LLM isn't seedable, so each is a fresh sample) and aggregates into <project>/bench/scorecard.md (per-seed rows + mean ± sample stddev for baseline/best/lift/test) and bench/sweep.jsonl. This is what turns a single noisy >-gated number into a benchmark — but it spends N× a run's tokens.

Commands: run (--dry-run = no-LLM preflight; --watch = re-run on a skill/config/target edit) · eval (--watch = cheap re-eval loop) · check (token-0 drift: inputs changed since last run?) · bench (--seeds N: k-seed variance scorecard) · adopt · deploy (--as skill|context, no tokens) · list · new (--local for a repo-local .skillsmith/) · init. Global flags: --home <dir>, --plain (disable color). Full options: --help.

How skillsmith differs from SkillOpt

skillsmith is an independent Rust reimplementation inspired by SkillOpt's core idea — treat the skill document as trainable state and accept edits only behind a held-out gate. It shares no code with SkillOpt and narrows the scope to one job: optimizing a skill against a real repo's tests. The crux is the oracle and target — SkillOpt scores against its own per-benchmark graders over a built-in suite; skillsmith points the same propose→gate discipline at your repo and gates on your own tests.

	SkillOpt	skillsmith
Language	Python	Rust
Gate signal	per-benchmark graders over a built-in suite (EM/F1 + execution checks)	runs your repo's own tests (verify_cmd) in a git worktree
Target	6 built-in benchmarks, 7 models, 3 harnesses	your own git repo — the agent edits files, your tests decide
Method	DL-style: epochs, batch, LR budget, slow/meta updates	lean greedy gate: propose → eval → keep strict lift → stage
Backends	OpenAI / Azure / Claude / Qwen / MiniMax (API keys)	claude / codex / gemini CLI (no key) or genai (key); per-stage model tiering
Cadence	batch training + SkillOpt-Sleep nightly offline self-evolution	explicit foreground runs + optional --watch dev loop (no background daemon, by design)
Surface	research framework + WebUI + PyPI	single-purpose local CLI

See the SkillOpt paper for the method and benchmarks behind the shared idea; SkillOpt-Sleep is its newer nightly self-evolution mode for local coding agents — skillsmith stays deliberately foreground (you run it; nothing auto-adopts).

Acknowledgements & prior art

The "skill-as-trainable-state + accept-only-behind-a-held-out-gate" framing comes from SkillOpt (Yang et al., 2026 — repo, paper; MIT, © Microsoft). skillsmith is an independent reimplementation that ports the idea, not the code — no SkillOpt source is in this repository, so it makes no "powered by SkillOpt" claim.

License

MIT © 2026 JHoo0118. SkillOpt is separately MIT-licensed by Microsoft; skillsmith includes none of its code, so no third-party notice is required.