Skip to content

BUILDLOG.md

Latest commit

 

History

History
829 lines (676 loc) · 55.4 KB

BUILDLOG.md

File metadata and controls

829 lines (676 loc) · 55.4 KB
Raw

Phoenix BUILDLOG — what worked and what didn't (honest, append-only)

Disregarding process ceremony by design. This is the real engineering record: dead ends, reversals, and surprises included. Failure that's recorded is progress; failure that's hidden is debt.

2026-06-09 — Day 0: grounding + mission

Goal: decide what Phoenix is, ground it in real references, write the mission, init the repo.

What worked

  • Located the real anchors on disk/web instead of guessing:
    • code/hermes = NousResearch Hermes Agent (Python): self-improving loop, autonomous skill creation, skills improve during use, agentskills.io-compatible. This is the "Hermes-like" ref.
    • agentskills.io = open SKILL.md standard + progressive disclosure (discover→activate→execute).
    • Addy Osmani agent-skills = lifecycle gates (spec/plan/build/test/review/ship) + anti-rationalization.
    • ATV-StarterKit (All-The-Vibes) = one-command Copilot setup, 4 pillars + Karpathy guardrails.
  • Confirmed toolchain: Rust 1.94.1 / cargo 1.94.1 present → Rust core is viable today.
  • Claude Code CLI confirmed as a driveable executor earlier this session (claude -p --output-format json).
  • Anchored Phoenix to the I2O result we actually measured (criteria-first verification), so the mission is grounded in evidence, not aspiration.

What didn't work / friction

  • ATV-StarterKit is NOT a local clone — it's a published npm installer (npx atv-starterkit). The goose-brain note on it was thin (one-line "memegen starter"). Had to pull the real repo from GitHub to learn the actual pillar architecture. Lesson: verify, don't trust the stub.
  • A PowerShell node -e "require('./package.json')" call hung (OneDrive/node interaction) and had to be killed. Lesson: prefer built-in view/glob tools over shelling node in the OneDrive tree.
  • Two tool calls got interrupted mid-flight (rapid user follow-ups). Re-ran cleanly. No harm.

Decisions

  • Language: Rust for the core (speed + inspectable + single binary). Skills stay portable markdown.
  • Standard: adopt agentskills.io, do not fork it. Phoenix skills must run on other compatible clients.
  • v0 scope locked (anti-astronomy): discover+load a skill → execute → SENSE outcome → ONE bounded self-heal (retry/rollback) → inspectable trace. Proven by a detected+recovered injected fault.

Open questions (carry forward)

  • How does Phoenix EXECUTE a skill's task — shell out to claude -p? call a local model? pluggable runner?
  • Trace format: JSONL event log (like our scorecard hash-chain) vs. structured spans?
  • Where does "sensing" get its signal for non-code tasks (exit code is easy; semantic outcomes are hard)?

Next step

Define v0 architecture (the spine: Skill loader · Runner · Sensor · Healer · Trace), then build the Cargo skeleton and the smallest end-to-end path that detects+recovers an injected fault.

2026-06-09 - Day 0 (cont.): community signal from All The Vibes "Hack & Furious" (last 10 days)

Goal: pressure-test the mission against what real practitioners are saying NOW.

What worked

  • WorkIQ surfaced high-signal, recent (last-10-days) quotes. Key grounding: the user's OWN published thesis repo All-The-Vibes/Agent-Harness ("Why the orchestration layer - not the model - determines agent success") with a canonical Five Pillars framework + an "attribution error" table.
  • Folded the Five Pillars into the mission as Phoenix's spine, and positioned Phoenix's two differentiators (self-healing + measured self-improvement) as explicit EXTENSIONS of them.
  • Promoted token efficiency to a first-class measured non-negotiable (tokens-per-verified-outcome), driven by recurring community signal that large skill sets burn enormous token budgets and that every token should earn its place in context. This also gives agentskills.io progressive disclosure a concrete Rust upgrade: a skill index with lazy/retrieval activation so only relevant skills enter context.
  • Sharpened positioning with the channel consensus that spec-kit/personas are being obsoleted in favor of context engineering (harness engineering reportedly burns far fewer tokens and feels closer to steering the agents). This is the real technical reason behind dropping ceremony.

Community signal (de-identified)

  • The harness is the chassis to the model's engine, and harness efficacy is benchmarkable.
  • When code fails, suspect/fix the harness before the code; every token should earn its place.
  • Specs-next-to-featureset and personas-as-behavioral-contracts are giving way to context engineering.
  • Large skill sets blow up token budgets at discovery time; harness engineering burns far less.
  • The owner's own Agent-Harness POV asked whether spec-kit and persona-based harnesses are being obsoleted.

What didn't work / friction

  • "SharkBait" (your earlier poor-man's-claude-code harness) had NO mentions in the last 10 days — it's older context. Noted as prior art to potentially mine, not current signal.
  • Parallelism / worktrees came up in the BROADER history (multi-agent consensus, many parallel agents, tmux panes) but NOT in the last-10-days channel window. Deferred: parallel/multi-agent orchestration is a real future theme but out of v0 scope. Recorded so we don't forget it.
  • One WorkIQ query got interrupted mid-flight (rapid follow-up); re-scoped to 10 days and it returned cleanly.

Decision

  • Phoenix = a Five-Pillar harness, in Rust, that self-heals and self-improves, and treats tokens as a measured budget. v0 scope UNCHANGED (load skill -> execute -> sense -> heal -> trace) but now the trace must also record token cost per step, so token-per-outcome is measurable from day one.

Next step

Design v0 architecture (Skill loader/index - Runner - Sensor - Healer - Trace[+token cost]) and scaffold the Cargo project via Claude Code CLI dynamic workflows.

2026-06-09 - Day 0 (cont. 2): TODAY's channel + correction

Goal: factor in conversations from TODAY specifically; user flagged key details.

What worked (today's signal, attributed)

  • Phoenix is already a live concept in-channel today - not just our internal name. Validated externally.
  • A channel participant anchored Phoenix to Hermes as the same underlying idea - confirms our grounding choice was right.
  • Another challenged whether this is just Agency or the existing GHCP CLI Plugin Marketplace -> Phoenix must EARN its existence vs prior art. Added a "reuse standards, don't reinvent" non-negotiable.
  • The owner reframed the ambition from "just agent behaviour" to an intelligent platform -> Elevated the mission: Phoenix is the harness done so well it becomes a platform (carries skills, senses, heals, improves, hosts an ecosystem across many runs).
  • An honest counter-signal questioned whether token costs matter on flat-rate plans -> Kept token-efficiency but RE-JUSTIFIED it as a latency/context-window budget (speed + room for the task), not primarily a $ cost argument. Honesty over hype.

What didn't work / correction

  • User directive: drop a specific community member's input entirely. I had folded two of their points into the mission (token-as-budget; fix the harness before the code) and their marketplace/add-ins framing was in the research notes. SCRUBBED all of that individual's fingerprints from MISSION.md (grep-verified zero remaining). Kept only the SUBSTANCE that stands on its own (token-as-budget; harness>model from the ATV POV white paper). Lesson: never capture third-party PII (names / verbatim quotes) into durable artifacts in the first place.

Net change to the mission

  • Thesis now: harness>model (ATV POV), elevated to intelligent platform (the owner's framing).
  • Must reuse agentskills.io + not reinvent Agency/GHCP-marketplace/Hermes (the prior-art challenge answered).
  • Token efficiency reframed as latency/context budget with the flat-rate caveat recorded honestly.

Still deferred (noted, not in v0)

  • Plugin/add-on marketplace + local-first runtime: real future themes raised today, but NOT v0. The v0 spine (load skill -> execute -> sense -> heal -> trace+tokens) is unchanged.

Next step

Design v0 architecture and scaffold the Cargo project via Claude Code CLI dynamic workflows.

2026-06-09 - Day 0 (cont. 3): ARCHITECTURE-DEFINING correction - Phoenix is a harness FOR GitHub Copilot

User directive: "this will be a harness for github copilot. just like how atv starter kit gets installed it will be installed in a similar fashion. we will use claude code to build out atv phoenix but the tool itself is built for github copilot."

What this corrected (big)

  • WRONG earlier model: Phoenix as a standalone Rust agent runtime that shells to an LLM (claude -p).
  • RIGHT model: Phoenix is an installable harness FOR GitHub Copilot (like ATV-StarterKit). The agent RUNTIME is GitHub Copilot (CLI + VS Code). Phoenix = the skills/agents/instructions/MCP layer that installs INTO Copilot. Claude Code is the BUILD tool (dev-time), not the product.

What worked (grounded via authoritative Copilot extensibility surface)

  • Copilot's real extension points (from Copilot CLI help): /skills (agentskills.io SKILL.md native), /agent, instructions files (copilot-instructions.md / AGENTS.md / .github/instructions/**), /mcp (MCP servers), /plugin marketplace. /env enumerates: instructions, MCP servers, skills, agents, plugins, LSPs, extensions.
  • Where Rust earns its place is now crisp: a fast Rust MCP SERVER that Copilot connects to via /mcp, exposing the capabilities Copilot lacks natively - objective SENSE, bounded HEAL, a token-cheap skill INDEX (lazy/retrieval), and an append-only TRACE with per-step token cost. Skills/ agents/instructions stay portable markdown; the spine is Rust. This resolves the "where does Rust fit" tension cleanly and keeps us standards-native (agentskills.io + MCP), answering the prior-art challenge.
  • Distribution mirrors ATV-StarterKit: copilot plugin marketplace add + px atv-phoenix init landing files in .github/ and ~/.copilot/.

v0 RE-CAST (Copilot-native, not standalone)

OLD: a cargo run binary that loads+executes a skill standalone. NEW: a Rust MCP server Copilot connects to, exposing sense/heal/trace tools; proven by a REAL Copilot session where an injected fault is sensed and a heal fires, shown in the trace. Same spine, correct host. v1 = measured skill self-improvement; v2 = one-command install of the full harness.

Open question now ANSWERED

  • "How does Phoenix execute a skill?" -> it doesn't; GitHub Copilot does. Phoenix senses/heals/traces around Copilot's execution via the MCP server + hooks/instructions. Local-first model choice is Copilot's /model, not ours.

Next step

Design the v0 Rust MCP server contract (sense/heal/trace tool schemas + trace JSONL format) and scaffold it with Claude Code CLI, then connect it to a live Copilot session and prove fault->heal.

2026-06-09 - Day 0 (cont. 4): two decisions - reuse Addy's skills (MIT) + graphify for retrieval

Q: "are we using addyosmani/agent-skills?" -> YES. Verified MIT license (c) 2025 Addy Osmani, so we can ship its 22 lifecycle skills + anti-rationalization gates DIRECTLY in Phoenix's starter skill pack (with attribution), not just as inspiration. Promoted in MISSION from "inspiration" to "reusable input." Policy added: bundle good MIT/Apache agentskills.io packs; build new only for the Rust spine.

Directive: "add code graph or graphify for token efficiency and fast retrieval." -> Adopted as the CONCRETE mechanism behind the previously hand-wavy "skill index."

  • graphify is already installed (~/.local/bin/graphify, "any input -> knowledge graph", outputs graphify-out/).
  • Two graphs: (1) SKILL graph - index every SKILL.md so the MCP skill_index tool returns only the relevant skill SUBGRAPH for the current intent (not all 50 -> kills the "45k tokens for 50 skills" problem); (2) CODE graph - graphify the target repo so retrieval pulls relevant symbols/callers/callees/blast-radius as a subgraph instead of dumping directories. Graph retrieval = structural (relationships) which flat embedding search misses. Directly serves Pillar 1 (Context Assembly) + tokens-per-verified-outcome.
  • Ownership: Rust MCP server owns graph build/query; graphify does extraction.

Net change

  • Pillar 1 extension is now "skill+code GRAPH (graphify)" not a vague "index."
  • New mission section "Token-efficient retrieval = a CODE GRAPH (graphify)".
  • Starter skill pack now explicitly includes Addy's MIT lifecycle skills.

v0 implication

  • v0 stays the sense/heal/trace MCP spine, but the FOURTH tool (skill_index via graphify subgraph query) is now concretely specified for v0.5/v1 rather than left abstract.

Next step

Design v0 Rust MCP server tool contracts (sense / heal / trace / skill_index) + trace JSONL format, scaffold via Claude Code CLI, prove fault->heal in a live Copilot session.

2026-06-09 - Day 0 (cont. 5): TokenMasterX is the token pillar - ALREADY BUILT + MEASURED

Q: "check shyamsridhar123/TokenMasterX" -> This is the user's OWN repo and it IS Phoenix's Context Assembly / token-efficiency pillar, already shipped and measured. Not inspiration - adopt directly.

What it is (decisive)

  • Routing agent for Claude Code + GitHub Copilot CLI (Phoenix's exact hosts), distributed the exact same way Phoenix will be: /plugin marketplace add shyamsridhar123/TokenMasterX + /plugin install token-master, then a per-repo /token-master command builds the graph and installs a routing agent to ~/.copilot/agents/token-master.agent.md with MCP servers declared inline.
  • Backend: graphify (default, no-LLM structural index) + codegraph (AST precise escalation).
  • Thesis = Phoenix's pillar verbatim: pay once to understand structure, then route structural questions ("who calls X", "what breaks if I change Y") to a prebuilt graph instead of grep-re-reading every turn.

Measured (use these, don't re-derive)

  • -73% cumulative input tokens, 3.71x overall, up to 7.8x on blast-radius, 12/12 from graph, 0 regressions - 36 live Copilot runs (scikit-learn + sympy). Honest negative reported (-44% one sympy inheritor; codegraph ~3-4x graphify on simple tasks). Optimizes AREA UNDER CONTEXT CURVE (cumulative tokens), not $ - matches our tokens-per-verified-outcome metric exactly.
  • Design lesson Phoenix inherits: ENFORCE routing, don't just offer (model used graph 0/15 unprompted, 8/8 when nudged). "A graph the model never queries saves nothing."

Strategic implication (big)

  • TokenMasterX is effectively a PROVEN single-pillar prototype of the Phoenix architecture - same hosts, same marketplace install, same MCP+routing-agent mechanism. One Phoenix pillar already ships with hard numbers. This massively de-risks the platform thesis and answers the "why not just X" challenge: we're composing a proven part, not reinventing.
  • Build strategy crystallized: Phoenix = COMPOSE proven parts (TokenMasterX + Addy MIT skills + agentskills.io) + BUILD only the novel spine (sense / heal / self-improve). Updated the "steal shamelessly" principle accordingly.

Net change to mission

  • Context Assembly pillar 1 = "adopt TokenMasterX" (was "Rust skill+code graph (graphify)").
  • New section replaces the graphify-from-scratch section with "adopt TokenMasterX."
  • Grounding now leads with TokenMasterX as the #1 reusable asset.

Re-scoped v0 (smaller + sharper)

  • v0 is now JUST the novel spine: a Rust MCP server exposing sense / heal / trace, proven by an injected fault detected+recovered in a live Copilot session. Token/retrieval is DONE (install TokenMasterX alongside). This makes v0 smaller and the whole thing more credible.

Next step

Design the v0 sense/heal/trace MCP tool contracts + trace JSONL format; scaffold via Claude Code CLI; prove fault->heal in a live Copilot session. Separately: try installing TokenMasterX into Copilot to validate the shared install path end-to-end.

2026-06-09 - Day 0 (cont. 6): MILESTONE M0 - install-path + token pillar VALIDATED (evals + screenshot)

User directive: "do it and we need clear evals and screenshots at every milestone." -> Stored as a standing Phoenix convention (evals/ + evals/screenshots/ per milestone). M0 is the first to honor it.

M0 result: PASS (objective)

  • Prereqs: uv 0.9.24, graphify (pkg 0.8.13), copilot CLI 1.0.61, node - all present.
  • TokenMasterX mechanism ALREADY installed on disk = the Phoenix install pattern, live: ~/.copilot/agents/token-master.agent.md (routing agent + inline graphify-nav MCP server: find/callers/callees/impact/inheritors/explain over .token-master/graph.json) + ~/.copilot/installed-plugins/_direct/token-master-plugin + anthropic-agent-skills packs.
  • Built a real graph: graphify update on ATV-Teams/packages/shared/src (TS) = 1322 nodes, 1717 edges, 70 communities in 10.7s, no LLM/API key.
  • Structural query answered FROM THE GRAPH and verified vs source: explain/query showed deriveAgentUrlKey() --calls--> normalizeAgentUrlKey(); ground-truth agent-url-key.ts:21 confirms it. CORRECT.
  • Evidence captured: evals/m0-install-path/RESULT.md + evals/screenshots/m0-graph-viz.png (1322-node interactive graph, headless-Chrome render). Screenshot visually verified (not blank).

What worked

  • The entire Phoenix install + token-retrieval pillar is REAL and already on this box -> massive de-risk.
  • graphify is fast + offline; correct structural answer on first try.

What didn't / friction (honest)

  • copilot CLI not on PATH (lives %APPDATA%/npm) - used absolute path. Document in install steps.
  • graphify skill 0.4.1 vs package 0.8.13 mismatch warning - cosmetic now, fix before relying on skill side.
  • kimi-webbridge daemon down -> headless Chrome (isolated profile) for the screenshot. Worked first try.

Decision

  • Token/retrieval pillar = VALIDATED, adopt TokenMasterX as-is. v0 scope NARROWS to the novel spine only (Rust MCP sense/heal/trace). No need to build any retrieval.

Next step

Design v0 Rust MCP server tool contracts (sense/heal/trace + trace JSONL), scaffold via Claude Code CLI, prove fault->heal in a live Copilot session -> that becomes Milestone M1 (with its own eval + screenshot).

2026-06-09 - Day 0 (cont. 7): MILESTONE M1 - self-healing spine BUILT + PASSING (evals + screenshot)

Goal: build the one novel thing - a Rust spine that senses objective failure and heals it.

Design-first + critique (worked)

  • Wrote docs/v0-spine-design.md, then ran a rubber-duck critique BEFORE coding. It caught a real flaw: the original demo (corrupt file -> restore -> check bytes==snapshot) is TAUTOLOGICAL (only proves restore restores). Adopted fix: success criterion = an EXTERNAL invariant (a real command exit code), not snapshot bytes. Also adopted: blessed snapshots (explicit snap_id, only snapshot a passing state), verify_trace as tamper-EVIDENT (not proof), stdout=JSON-RPC-only rule, argv-only command exec, pin rmcp.

Built (Rust lib phoenix)

  • sense (command_exit/file_sha256/regex_in_file, no LLM), snapshot (bless-only-if-check-passes, atomic restore), heal (rollback/retry, bounded <=3, healed only if EXTERNAL recheck passes), trace (append-only hash-chained JSONL, verify() tamper-evident - same scheme as goose scorecard). phoenix-mcp bin = stub.

Evals: PASS

  • cargo test = 3/3 green: green_red_heal_green_with_trace (behavioral, external signal), trace_is_tamper_evident (edits caught at broken_at=0), snapshot_refuses_to_bless_bad_state.
  • cargo run --example demo_self_heal emits a verified 4-row trace: sense(ok) -> snapshot(blessed) -> sense(RED) -> heal(healed) ; trace.verify ok=true rows=4.
  • Evidence: evals/m1-self-heal/RESULT.md + evals/screenshots/m1-self-heal.png (visually verified).

What didn't / friction (honest)

  • First demo design was tautological - caught by critique before coding (cheapest possible fix).
  • command_exit timeout documented but not yet in-process enforced (v0 limit; harden before live Copilot).
  • rmcp/stdio MCP wiring DEFERRED to M2 - spine proven independently of protocol churn first.

Scope honesty

  • Called "bounded objective recovery," NOT broad self-healing. Rollback is one strategy; no diagnosis claimed.

Next (M2)

Wire spine into a LIVE Copilot session via rmcp stdio MCP + /mcp; prove sense+heal from inside Copilot (not just cargo test). Eval + screenshot of the live session.

2026-06-09 - Day 0 (cont. 8): MILESTONE M2 - spine works over REAL MCP (evals + screenshot)

Goal: make the M1 spine callable by GitHub Copilot via an rmcp stdio MCP server; prove sense+heal THROUGH the protocol.

Built

  • src/bin/phoenix_mcp.rs: rmcp 1.7 stdio MCP server, 4 tools (phoenix_sense/snapshot/heal/verify_trace), thin adapters over the M1 lib. stdout=JSON-RPC only, diagnostics->stderr.
  • tests/m2_mcp_session.rs: spawns the server, does the MCP handshake, drives the full self-heal flow over real JSON-RPC with the fault injected MID-SESSION (one process = consistent trace).

Evals: PASS

  • initialize -> protocol 2025-06-18 + tools capability; tools/list -> 4 tools with full JSON schemas.
  • Full chain over MCP: sense(GREEN) -> snapshot(blessed) -> [inject fault] -> sense(RED) -> heal(healed=true) -> sense(GREEN) -> verify_trace(ok, rows>=5).
  • cargo test = 4/4 (3 spine + 1 MCP-session). Evidence: evals/m2-mcp/RESULT.md + session.txt + evals/screenshots/m2-mcp-session.png (visually verified).

What worked

  • rmcp macros generated correct MCP tool schemas from Rust types - Copilot gets typed contracts free.
  • M1 lib dropped straight behind the MCP adapter (compose, don't rewrite).

What didn't / friction (integration testing earned its keep)

  • REAL BUG caught only by end-to-end MCP testing: heal(rollback) resolved ctx.path against process CWD not the workspace -> first run healed the wrong file (healed=false). Fixed: workspace.join(path) (no-op for absolute paths, so M1 still passes). A library-only test would have missed this.
  • First harness drove TWO server processes (to inject fault between calls) -> fragmented trace + buggy PowerShell parsing. Replaced with a single-process Rust integration test. Lesson: one stdio session.
  • ServerInfo is #[non_exhaustive] -> build via default()+assign, not struct literal.

Scope honesty

  • Proven against a Copilot-LIKE MCP client (the integration test). Driving from the actual interactive copilot CLI (install agent + /mcp) is a thin remaining step (M3), not a code risk - protocol proven here.

Next (M3)

Package as an installable Copilot plugin (agent def + mcp-servers block + npx/marketplace, ATV-StarterKit style) and drive a real fault->heal from inside an interactive copilot session. Eval + screenshot.

2026-06-09 - Day 0 (cont. 9): MILESTONE M3 - LIVE self-heal inside the real GitHub Copilot CLI

THE milestone: Copilot itself (copilot -p, v1.0.61) called Phoenix's MCP tools to sense + heal a real fault, file fixed on disk, verified trace. Not a test harness - the actual product runtime.

Install mechanism

  • dist/phoenix.agent.md (Copilot agent def, token-master pattern) + dist/install.ps1 (ATV-StarterKit style).
  • What worked for LIVE invocation: registering phoenix in ~/.copilot/mcp-config.json (the user MCP registry). Copilot auto-discovered phoenix_sense/snapshot/heal/verify_trace.

Live proof

  • Prompt: "use phoenix tools to verify+recover logic.txt". Copilot's calls: phoenix_sense -> ok:false (RED) ; phoenix_heal -> healed:true (rollback) ; phoenix_sense -> ok:true (GREEN). Copilot reported Before:false After:true. File on disk = answer=GOOD_MARKER (fixed). ~14.9 credits/187.6k tok/36s.
  • Trace written DURING the live session (evals/m3-live-copilot/live-trace.jsonl): 3 hash-chained rows, GENESIS->0f2254c6->a67e1126->6ec26de8.
  • Evidence: evals/m3-live-copilot/RESULT.md + live-trace.jsonl + evals/screenshots/m3-live-copilot.png.

Bug caught ONLY by the live LLM (and fixed)

  • Copilot passes expect:0 as an INTEGER; schema wanted string -> first sense call errored -32602. Copilot self-corrected but wasted a turn. FIXED: expect now accepts string|number|null via custom deserializer de_string_or_number + unit test. RE-VERIFIED LIVE: integer expect now accepted cleanly. This is the class of bug no lib/protocol test finds - only a real model calling the tool.

Friction (honest)

  • Loose ~/.copilot/agents/phoenix.agent.md NOT picked up by --agent phoenix (Copilot resolves --agent against registered/installed-plugin agents). Used mcp-config registry for the live proof. Proper marketplace/npx plugin registration (so --agent phoenix works) is the remaining packaging step.
  • copilot -p headless is slow (30s-min+); inline --additional-mcp-config JSON gets mangled by the PowerShell->cmd shim -> use the config-file path.

Tests: cargo test 5/5 (expect-flex + 3 spine + 1 MCP-session), 0 regressions.

Next (M4+)

  • Proper plugin packaging so copilot --agent phoenix works (one-command install).
  • Harder live scenario: Copilot edits real code, breaks a real test, self-heals - measured.

2026-06-09 - Day 0 (cont. 10): H2 EXPERIMENT - objective verifier vs vanilla, LIVE on Copilot (20 sessions)

Goal: Does giving real Copilot the phoenix_sense+heal loop change outcomes vs vanilla self-judgment? (H2 + Phoenix's core product claim, on the real runtime.)

Result: POSITIVE where it matters

  • WELL-SPECIFIED (slugify/duration/roman): vanilla 6/6 = phoenix 6/6 (CEILING - strong model, no harm).
  • UNDERSPECIFIED (clamp/initials, spec omits a checker-enforced criterion): vanilla 0/4 with 4 SILENT FAILURES (claimed DONE every time, checker failed every time); phoenix 4/4, 0 silent failures.
  • OVERALL n=10/arm: silent-failure 40%->0%, verified-pass 60%->100%, 0 regressions. Phoenix tools used 10/10 phoenix runs, 0/10 vanilla (enforce-vs-offer replicated).
  • Evidence: evals/h2-experiment/RESULT.md + results_all.jsonl + evals/screenshots/h2-results.png.

What worked

  • Clean two-arm design scored by EXTERNAL hidden checkers (ground truth), pre-flighted (accept-good/ reject-naive) before spending credits. Adversarial underspecified tasks isolated the H2 signal cleanly.
  • Phoenix reliably invoked by live Copilot in every B run; healed every underspecified failure.

What didn't / friction (a LOT of harness debugging - documented honestly)

  • copilot needs --add-dir to edit files in a temp dir (silent no-op without it).
  • Start-Job AND Start-Process both mangle the copilot.cmd arg quoting -> use the direct & \ call op.
  • PowerShell case-INSENSITIVE vars: \ (path) collided with \ (array) -> renamed .
  • Specs with -> arrows / quotes / dash-led tokens get parsed as CLI options by the cmd shim -> rewrote specs as plain prose + sanitize (strip quotes, collapse ws).
  • First well-specified round was a CEILING null (both 100%) - same lesson as H1; added adversarial underspecified tasks to actually exercise the silent-failure-catch the experiment was built for.

Honest scope

  • 2 reps/task, single model, deterministic checkers, constructed underspecified tasks. Directional signal with an unambiguous mechanism; replicate at larger n before a hard statistical claim. Report BOTH the ceiling and the positive - no cherry-picking.

Product implication (M4)

  • Headline for shippable Phoenix: "cut Copilot silent-failure 40%->0% on hidden-criteria tasks, 0 regressions."

Next

  • M4 product packaging around this number; H1 replication still owed to the heartbeat; Scout adapter spike.

2026-06-09 - Day 0 (cont. 11): M4 SHIP v0.1.0 + Scout spike + multi-host CLI

M4: Phoenix is now a shippable product.

  • Product README (leads with the H2 number: 40%->0% silent failures), CHANGELOG, v0.1.0 tag pushed.
  • Added CLI MODE to phoenix-mcp: phoenix-mcp sense|snapshot|heal|verify-trace '' with pass/fail exit codes. SAME binary now serves Copilot (MCP, no args) AND Scout (shell CLI subcommands). cargo test 5/5.

Scout adapter spike (honest feasibility):

  • FINDING: Scout does NOT accept arbitrary external MCP servers - its server set is fixed/builtin (filesystem, playwright, shell, workiq), confirmed in ~/.copilot/m-mcp-servers.json + m-settings.json.
  • BUT Scout has a shell tool + a skills system. So the Scout adapter = the phoenix-mcp CLI (called via shell, exit code = pass/fail) + dist/scout/phoenix-self-heal.skill.md (teaches the verify-heal loop).
  • One Rust core, two adapters (MCP for Copilot, CLI for Scout) = the multi-host promise, real. Validated the CLI end-to-end: sense GREEN exit0, sense RED exit1, snapshot blessed, heal rollback healed, verify-trace ok 5 rows.

What worked: the host-agnostic core paid off - shipping for a second host was a CLI dispatch + a skill file, not a rewrite. README anchored to a real measured number, not aspiration.

What didn't / honest: --agent phoenix still needs marketplace/plugin registration (live use is via mcp-config registration); deferred to a later packaging pass. command_exit timeout still not in-process.

Next: H1 replication (running off-peak in background) to be recorded when done.

2026-06-10 - Day 1: one-command install + dogfooding (UX fix) + H3 memory-lift

(1) One-command install: built .copilot-plugin/ (marketplace.json + phoenix-setup SKILL.md + setup.py). setup.py is idempotent: finds repo, builds binary if missing, registers the phoenix MCP server in ~/.copilot/mcp-config.json, installs the agent. Validated: clean install + live heal works.

(2) Dogfooding caught a REAL UX failure (the value of dogfooding): first live run cost 72 AI credits / 4 min / ~25 FAILED tool calls - Copilot kept guessing the strict sense schema wrong (missing kind/target, target-as-string, expect-as-int, bare argv without cmd /C). FIX: made sense inputs lenient (target accepts string|array via de_string_or_vec; expect accepts int|string|null) + put an explicit EXAMPLE in the tool description. RE-MEASURED: 14.8 credits / 52s / 4 calls = ~5x better. cargo test 6/6. Also fixed setup.py unicode crash on Windows console (ascii output).

(3) H3 memory-lift: POSITIVE, decisive. Tasks whose correct answer depends on a project convention the model can't guess (pre-flighted: default solution FAILS the hidden checker). A_nocontext (spec only) 0/4; B_context (spec + injected convention) 4/4 = 0%->100%. Without context Copilot produced the standard default (Active, ,234.50) every time; with it, followed the convention exactly. Evidence: evals/h3-experiment/RESULT.md + results.jsonl + evals/screenshots/h3-results.png.

What didn't / honest

  • userid (3rd H3 family) abandoned mid-run to API rate-limiting; 8 completed trials show a perfect 0/4 vs 4/4 split so the conclusion is not in doubt, but it is 8 trials not 12.
  • --agent phoenix resolution still depends on agent-file registration; the plugin/marketplace path is scaffolded (.copilot-plugin/) but copilot plugin install end-to-end not yet verified.

The trio is complete

H1 (criteria-first, replicated +0.125) + H2 (objective verify, 40%->0% silent fail) + H3 (context, 0->100%) = formalize intent + verify objectively + supply the right context. The I2O thesis, measured.

2026-06-10 - Day 1 (cont.): BUNDLED self-maintaining skill pack (user: "it has to be bundled and self-maintaining")

User push (correct): a self-healing harness can't depend on you manually installing other plugins. It must be SELF-CONTAINED (bundled) and SELF-MAINTAINING. Chose: rebuild a lifecycle pack GROUND-UP, Phoenix-native, rather than copy Addy's verbatim - because Phoenix's differentiator (objective verification gate) doesn't exist in generic workflow skills.

Built (bundled, ships in repo skills/):

  • 6 agentskills.io skills: phoenix-spec, phoenix-plan, phoenix-build, phoenix-review, phoenix-ship, phoenix-self-heal. EVERY lifecycle stage is gated by an objective phoenix_sense check (spec's deliverable is a runnable check; build advances only on green; ship refuses to declare done without a green sense + verified trace). This is the thing Addy's pack doesn't have.

Self-maintaining mechanism (the real meaning of self-healing):

  • src/doctor.rs + phoenix-mcp doctor: Phoenix validates its OWN bundled skills (frontmatter present, name matches dir, description non-trivial) using its own evidence discipline. 6/6 OK live.
  • tests/skills_doctor.rs: cargo test FAILS if any bundled skill drifts -> the harness catches its own rot objectively, in CI. cargo test now 8 tests (added 2).
  • setup.py now INSTALLS all bundled skills to ~/.copilot/skills AND runs doctor as a post-install self-check. Verified end-to-end: 6 skills installed, doctor OK.

README: honest stack table - lifecycle pack is BUNDLED (auto-installed, self-maintaining); TokenMasterX is a detected recommended companion; Addy's pack optional.

What this fixes: the repo previously shipped ZERO bundled skills while the README claimed to compose them. Now it ships its own verification-gated lifecycle pack that it can verify and heal itself.

2026-06-10 - Day 1 (cont. 2): BUNDLE TokenMasterX (user: "bundle it because it is mine")

  • TokenMasterX is the user's OWN MIT plugin (c) 2026 Shyam Sridhar - so vendoring it makes Phoenix truly self-contained (NOT the reinvent-someone-elses-wheel anti-pattern). Vendored to vendor/token-master/ (plugin.json + SKILL.md + graphify_mcp.py + setup.py + agent templates + LICENSE).
  • setup.py now INSTALLS the bundled TokenMasterX (invokes its setup.py --host=copilot) instead of just recommending it. Verified end-to-end: token-master agent installed + graph built on the phoenix repo.
  • One command now installs the FULL stack: lifecycle skills + self-heal MCP + bundled TokenMasterX.
  • README stack table: TokenMasterX now BUNDLED (needs graphify); only Addy's general pack stays optional.
  • .gitignore: vendor pycache excluded; .token-master/ graph already ignored.

Net: Phoenix is self-contained (skills + token retrieval bundled) AND self-maintaining (doctor + cargo test).

2026-06-10 - Day 1 (cont. 3): COMPREHENSIVE skill pack (user: "dont half-ass it, as rich as Addy's")

Studied addyosmani/agent-skills in depth (cloned: 22 skills, 124-363 lines each, signature devices = ASCII flow diagrams + concrete examples + 'Common Rationalizations' tables + 'Red Flags' sections + routing decision tree in a meta-skill). My first pass (25-130 line stubs) was half-assed; rebuilt.

Rebuilt as 10 comprehensive skills (each: overview, when-to-use/not, ASCII diagram, worked examples, rules, Common Rationalizations table, Red Flags, Next):

  • phoenix (META ROUTER): routing decision tree + 6 non-negotiable Phoenix Laws + untrusted-error-data rule.
  • phoenix-think: deep interview + deep research -> Intent Contract w/ acceptance check (renamed from spec per user; first stage is THINK not spec).
  • phoenix-plan, phoenix-build, phoenix-test (TDD w/ sense as gate), phoenix-debug (triage+heal), phoenix-context (TokenMasterX graph routing = the token-efficiency skill), phoenix-review, phoenix-ship, phoenix-self-heal.

The differentiators woven through EVERY skill (this is what makes it Phoenix, not a copy of Addy):

  • every gate is an objective phoenix_sense check (Addy's are workflow prose, no objective gate)
  • self-healing: snapshot before risk, bounded heal on red, external recheck confirms
  • token-efficient: route structural Qs to the graph not grep; pull subgraphs not directories; progressive disclosure; "area under the context curve" framing in phoenix-context.

Self-maintaining: phoenix-mcp doctor validates all 10 (name==dir, frontmatter, desc); cargo test asserts >=10 valid + catches drift. Verified: 10/10 OK, full suite green, install ships 10 + self-checks.

2026-06-10 - Day 1 (cont. 4): incorporate Karpathy + Mat Pocock + Emil Kowalski craft skills

User: incorporate the lifecycle pack WITH karpathy, mat pocock, and emil kowalski design skills. Grounded each in the real source (karpathy-guidelines + emil-design-eng skills read locally; Mat Pocock / Total TypeScript fetched from totaltypescript.com), then built Phoenix-native versions where each master's discipline becomes an OBJECTIVE GATE:

  • phoenix-craft (Andrej Karpathy): think-before-coding, simplicity-first, surgical changes, verifiable success - but "simpler"/"done" must re-pass phoenix_sense (craft proven, not asserted). Attributed + linked.
  • phoenix-typescript (Mat Pocock / Total TypeScript): sc --noEmit IS a phoenix_sense check; strict mode, kill any/unsafe casts, derive don't duplicate types, no @ts-ignore-to-pass (= editing the test). The Pocock move: change the type first, let tsc show you the blast radius.
  • phoenix-design (Emil Kowalski): the Animation Decision Framework (should it animate / purpose / curve), the REQUIRED Before/After review table, ease-out + :active + reduced-motion - gated by build/lint/visual sense so polish ships without regressing. Links animations.dev.

Each is comprehensive (overview, framework/tables, examples, Common Rationalizations, Red Flags, Next) and credits the source. Meta-router updated to route to them. Pack now 13 skills.

Self-maintaining holds: doctor 13/13 OK; cargo test asserts >=13 + catches drift; full suite green; one-command install ships 13 + self-checks. README + setup.py updated.

2026-06-10 - Day 1 (cont. 5): END-TO-END sandbox test (fresh clone -> install -> real game build)

User: don't just test a broken file, do an E2E build of a real project like a space invaders html game.

Did exactly that in an ISOLATED sandbox (code/phoenix-e2e-sandbox):

  1. Fresh git clone from the remote -> 13 skills + vendored TokenMasterX present.
  2. Cold cargo build --release (110s) + cargo test -> 6/6 suites PASS (shipped repo is green).
  3. setup.py into a FAKE HOME (real ~/.copilot untouched) -> MCP + agent + 13 skills + doctor 13/13 OK.
  4. REAL BUILD: live Copilot built a complete Space Invaders game.html (254 lines/8164 bytes) under the Phoenix loop, gated by an objective node check.js (9 mechanics + size). phoenix_sense ok:true, phoenix_verify_trace ok:true, independent recheck exit 0. 23.6 credits / 1m10s.
  5. Screenshot proves it RENDERS + runs (5 rows invaders, player ship, score, arcade aesthetic).

Evidence: evals/e2e-sandbox/RESULT.md + space-invaders-game.html + game-check.js + evals/screenshots/e2e-space-invaders.png.

Friction (honest): --additional-mcp-config takes inline JSON not a file path, and the PowerShell->cmd shim mangles inline JSON -> used the proper mcp-config registration path (what a real user does) instead. Restored real mcp-config afterward. The isolated-HOME install proves the path is clean + portable.

This is the strongest evidence yet: a stranger could clone, one-command install, and have Copilot build

  • objectively verify a real game. The whole product works end to end.

== 2026-06-10 (cont.) :: SWE-bench-style lite benchmark == WHY: "can we do a swebench or some lite version of a legit benchmarking tool". Needed a citable, rigged-proof resolved-rate number, not a vibe. WHAT: borrowed the SWE-bench EVALUATION CONTRACT (the part that earns trust) without the official Docker/repo machinery: RESOLVED iff all FAIL_TO_PASS go green AND all PASS_TO_PASS stay green. Every task pre-validated TWO ways before any agent ran: (a) buggy state = F2P red + P2P green (a real bug), (b) solvable = a reference fix turns F2P green keeping P2P green (no impossible/rigged tasks). Two arms: A_vanilla (problem.md only, tests HIDDEN, self-judges) vs B_phoenix (same + tests as the enforced phoenix_sense gate). Confound stated openly: B seeing the tests IS the intervention.

RESULT (9 tasks x 2 arms, Copilot CLI 1.0.61):

  • Tier 1 well-specified (5): vanilla 100% / phoenix 100%. CEILING -- Phoenix adds no lift AND no harm. Honest null, same regime the I2O H1 nulls located. A benchmark that only showed this would be boring but true.
  • Tier 2 underspecified (4, terse GitHub-issue style, hidden edge cases): vanilla 50% / phoenix 100%. 2 tasks (hard-slugify, hard-titlecase) flipped unresolved->resolved.
  • Overall: 78% -> 100%, +22 pts, 0 regressions in EITHER arm. KEY: both vanilla misses were SILENT FAILURES not crashes (f2p=0, p2p=1) -- the happy path passed, the hidden edge didn't; the fix LOOKED done. Same failure mode H2 measured, now under a SWE-bench contract. An enforced objective gate converts "looks done" into "is done."

HONEST LIMITS: n=9, 1 rep, 1 model = directional. SWE-bench-STYLE (self-contained tasks), not the official dataset. The headline lift is entirely from the underspecified tier -- which is exactly where an objective gate is supposed to pay off; on clear specs the model already wins. PROCESS NOTE: first run was the well-specified tier alone = 100/100 ceiling. Rather than ship a null or juice it, added a designed underspecified tier to LOCATE the regime where the gate matters (the H1 null->null->positive playbook). Added -Filter/-OutFile/-Append to run_swe.ps1 so tiers run without re-spending calls. Evidence: evals/swe-bench-lite/RESULT.md + results.jsonl (+tier1/tier2) + run_swe.ps1 + tasks/ + evals/screenshots/swe-bench-result.png.

== 2026-06-10 (cont.) :: autonomous workflows — ralph / goal / dynamic, gated by proof == WHY: user asked to implement /phoenix-ralph /phoenix-goal /phoenix dynamic workflows — "the same capabilities available in claude code" (ralph loop, goal, dynamic workflows). RESEARCH: a research subagent compiled a sourced report (research/autonomous-workflows-research.md) from PRIMARY sources — Geoffrey Huntley's Ralph (ghuntley.com/ralph: while-loop, fresh context per iteration, filesystem as brain via fix_plan.md+AGENT.md, one task per loop, no auto-stop), Anthropic's Building Effective Agents (routing/orchestrator-workers/evaluator-optimizer) + SWE-bench scaffold, BabyAGI, ReAct. Also read the LIVE OMC ralph/autopilot/ultrawork/ralplan SKILL.md on this box — their final gate is an LLM reviewer's OPINION. That's the gap Phoenix closes.

RUBBER-DUCK (before building): caught the core flaw — my first design trusted an agent-authored backlog done:true + prose "reproduce-first" rules. Fix adopted: completion must be DERIVED by the driver from the trace, failure-first, enforced in TOOLING not prose. Also: driver (not agent) owns sentinel/tag/budgets; top-level gate must be command_exit; ship dynamic routing as a SEPARATE skill so the fixed phoenix tree doesn't regress. These shaped the final build.

BUILT:

  • Gate ledger (src/accept.rs, phoenix-mcp accept): ok ONLY if trace shows the exact check (by new canonical_digest) went red→green, chain intact, green now. This is the centerpiece — it makes a vacuous check (test -f) provably worthless: a gate never seen red is rejected. tests/gate_ledger.rs proves accept(red→green)=ok, accept(never-red)=reject, accept(tampered)=reject. All green.
  • canonical_digest(&Check): MCP digested only target; CLI digested raw arg string — inconsistent, so a check wasn't identifiable across the trace. Now both record canonical_digest (kind+target+normalized expect), so "this exact check went red then green" is provable.
  • 3 skills (pack 13→16): phoenix-ralph (persistence loop), phoenix-goal (formalize an objective acceptance check FIRST, then decompose+drive), phoenix-auto (dynamic router, opt-in; base phoenix tree unchanged). doctor 16/16; skills_doctor floor bumped to >=16.
  • dist/ralph/phoenix-ralph.ps1 (+ .sh): external loop driver. Driver decides done via accept, owns MaxLoops/MaxMinutes/NoProgressStop, verify-trace each iter, writes completed.json + git tag. Agent only proposes. FRICTION (honest):
  • PowerShell→exe quote-mangling corrupted inline JSON ("key must be a string") — same class as the earlier --additional-mcp-config pain. Fixed properly with an @file arg convention in the CLI; the driver passes @done-check.json. Robust, no escaping hell.
  • Driver crashed on git tag in a commit-less test repo ($ErrorActionPreference=Stop turned git stderr into a terminating error, flipping a PROVEN completion to exit 1). Made tagging best-effort (try/catch, never undoes a proven accept).
  • A registered phoenix-mcp.exe (the real mcp-config server) locked the binary → build "Access denied"; stopped PID, rebuilt. EVIDENCE (no real Copilot spend — deterministic): gate_ledger 3/3; CLI accept rejects vacuous (exit 1) / accepts red→green (exit 0); driver Case A refuses an already-green done-check (exit 2); driver Case B happy path proves completion + writes completed.json + tags (exit 0). Screenshot: evals/screenshots/autonomous-workflows.png. A live end-to-end Copilot build under the loop is the next evidence step. KEY INSIGHT: this is the Intent-to-Outcome loop made operational — ralph is ACT under persistence, goal is FORMALIZE→ACT→VERIFY, and the gate ledger is VERIFY turned into machine-enforced proof. Every other autonomous loop ends in an opinion; Phoenix ends in evidence.

2026-06-17 — OKF: knowledge as portable, inspectable bundles (phoenix-okf skill)

Goal: implement the Open Knowledge Format (OKF v0.1, GoogleCloudPlatform/knowledge-catalog) for Phoenix — produce, validate, and consume vendor-neutral markdown+frontmatter knowledge bundles. Three slices in one skill: skills/phoenix-okf.

What worked

  • Slice 1 — code-graph exporter (scripts/okf_export.py): .token-master/graph.json → OKF bundle, one concept doc per source file, cross-file edges as bundle-relative markdown links. INFERRED edges flagged candidate (honors phoenix-context's ~0.8-confidence honesty rule). Auto-writes per-dir index.md, a root index.md (declares okf_version), and log.md. Ran on Phoenix's own graph → 50 conformant concepts from 445 nodes / 508 edges.
  • Validator as the phoenix_sense gate (scripts/okf_validate.py): enforces §9 (parseable frontmatter + non-empty type; reserved-file rules). Exit 0 = conformant. Both bundles pass, 0 broken links.
  • Slice 2 — skills as an OKF bundle (scripts/okf_skillsync.py): surgical idempotent insert of type: Phoenix Skill into all 16 SKILL.md (non-breaking; agentskills ignores unknown keys) + generated skills/index.md. skills/ now validates CONFORMANT (17 Skill + 1 Reference).
  • Slice 3 — consumer (scripts/okf_ingest.py): index-first progressive disclosure (types, tags, concept list) so a run pays once for orientation; --query by type/tag, --full for one body. Permissive per §9 (tolerates unknown types / broken links).

What didn't work / friction

  • First skills/ validation FAILED: the bundled spec mirror OKF-SPEC-0.1.md had no frontmatter (→ §9.1 error) and its example snippets produced 11 broken-link warnings. Fixes: (a) prepended type: Reference frontmatter to the mirror; (b) taught the validator to strip fenced code blocks before scanning links (illustrative links in examples are not real edges). Re-validated clean.
  • Console Get-Content piping visually merged the closing --- with the next line; the view tool confirmed the files were correctly delimited. Lesson: trust the parser/view, not the pager.

Decisions

  • Granularity = one concept per source file (diffable, human-meaningful) — node-level detail lives in the body; the graph lives in cross-file links. 50 docs, not 445 noise docs.
  • Committed examples/okf-code-graph/ as a living reference bundle (OKF philosophy: bundles live in git, reviewable), even though .token-master/graph.json itself is gitignored.

EVIDENCE (deterministic, no Copilot spend): okf_validate examples/okf-code-graph → CONFORMANT (50 concepts, 0 broken links); okf_validate skills → CONFORMANT (18 concepts); okf_ingest round-trips both bundles (outline + type/tag query + --full retrieval). KEY INSIGHT: Phoenix's strongest measured pillar (TokenMasterX graph) stops hiding in JSON — its knowledge becomes a git-reviewable, any-tool-readable artifact, and producing + consuming OKF is the concrete step from "code harness" to "knowledge platform."

2026-06-17 (later) — M4: OKF self-sensing + measured (sense/freshness in the spine, token eval)

Goal: make OKF (1) self-sensing via the Phoenix spine and (2) backed by real token numbers.

What worked

  • No Rust change needed. The spine's existing command_exit sense kind runs the Python OKF gates — OKF domain logic stays in portable skill scripts, spine stays minimal. Right call.
  • Freshness sense (scripts/okf_freshness.py): compares built_at_commit anchored in the bundle root index.md vs the live graph.json. Proven FRESH (exit 0) and STALE (exit 1, vs a fabricated commit). Export now writes the commit anchor + generated_at into the root index.
  • Full red->green through the REAL phoenix-mcp.exe: sense conformance GREEN -> inject fault (strip a concept's type) -> sense RED -> heal (re-export) -> sense GREEN -> accept proves failure-first red->green in an intact trace (rows=4, chain verified). Same M1/M2 self-heal pattern, now applied to knowledge artifacts.
  • Committed sense recipes (skills/phoenix-okf/checks/*.json) so any host can pass them straight to phoenix_sense/phoenix_accept.
  • Token eval (evals/m4-okf/, tiktoken o200k_base, deterministic): index-first vs raw graph.json = 31.3x fewer tokens single-shot; vs whole-bundle 4.4x; session view (outline paid once) index-first 4,113 tok beats raw 90,067 (21.9x) / whole 17,911 (4.4x) / grep 8,778 (2.1x).

What didn't work / friction

  • First pass charged the 2,470-tok outline to EVERY question, making single-shot grep look cheaper (8,778 vs 11,523). Honest finding kept AND fixed with a session-amortized view (outline once) where index-first wins 2.1x over grep. Reported the grep nuance rather than hiding it.
  • Adding checks/README.md briefly broke skills/ conformance (non-reserved .md needs type); fixed with type: Reference frontmatter. Lesson: every .md added under a bundle is a concept.
  • Validator stdout used x/S-unsafe unicode (×, §) that mojibake'd in the trace evidence; switched to ASCII so the trace reads clean.

Decisions

  • Spine stays minimal — no OKF-specific Rust CheckKind; command_exit + Python gates is the composition-over-bloat choice consistent with the mission.
  • Report the honest negative (small-bundle single-shot grep is competitive) — credibility over spin, matching the TokenMasterX precedent.

EVIDENCE (deterministic, no Copilot spend): both bundles CONFORMANT; freshness FRESH/STALE both proven; phoenix-mcp sense/accept/verify-trace chain green with red->green; evals/m4-okf/RESULT.md + results.jsonl. KEY INSIGHT: a knowledge bundle that drifts or corrupts now emits an objective RED a run can heal — knowledge becomes a first-class sensed, self-healing artifact, not a static dump.

2026-06-17 (later still) — M5: OKF consumed, interop-proven, tested, CI-gated + a community demo

Goal: finish the OKF arc — prove Phoenix consumes OKF in a live phoenix-context run (#3), interops with a foreign bundle (#4), and lock it all behind tests + CI (#5), then package a demo the community can run.

What worked

  • #3 Live consumption (evals/m5-okf-live/): a real phoenix-context turn loop answers a genuine structural question ("what cross-file edges does src/heal.rs have?") by invoking okf_ingest as a subprocess on the committed bundle — index-first outline -> --query -> open exactly one concept. Honest result: single isolated turn, grep-and-read is 1.3x cheaper on a 50-file bundle; the session/multi-turn view (outline paid once) flips it to OKF. Reported, not hidden. Wired okf_ingest into skills/phoenix-context/SKILL.md as the OKF consumption path.
  • #4 Interop (examples/okf-external-demo/): a hand-authored bundle from a fictional acme-knowledge-catalog with a NON-Phoenix vocabulary (Runbook/Dataset/Decision/Glossary) is CONFORMANT even under --strict-links (0 broken links) and ingests index-first. Proves the gate and consumer are vendor-neutral — no Phoenix-specific assumptions baked in.
  • #5 Tests + CI: tests/okf/test_okf.py (12 pytest cases: export->validate->freshness->ingest round-trip on a synthetic graph, validator catches missing-type and index-with-frontmatter, both committed bundles conformant, interop). tests/okf_sense.rs (3 cases): the spine itself senses both committed bundles GREEN and a broken bundle RED via command_exit — same self- maintenance discipline as skills_doctor.rs, now for knowledge. First CI for the repo: .github/workflows/okf.yml (python gate + cargo test --test okf_sense).
  • Community demo (demo/okf/run-demo.ps1): one non-destructive, narrated, runnable script that walks PRODUCE -> VALIDATE -> SENSE+HEAL (through the real phoenix-mcp.exe: green->RED->heal-> green, accept ok=true, trace rows=5 verified) -> CONSUME -> INTEROP. Plus demo/okf/README.md with a per-beat talking track. Verified end-to-end.

What didn't work / friction

  • PowerShell 5.1 wrote check.json/healctx.json with a UTF-8 BOM, which serde rejected ("expected value, line 1 column 1"). Switched to [IO.File]::WriteAllText (no BOM). Also dropped all non-ASCII (em-dashes) from the script after PS 5.1 ANSI-decoded the file and broke parsing.
  • A pytest asserted okf_version == "0.1" but PyYAML reads 0.1 as a float; normalized with str(...). Lesson: unquoted YAML scalars are typed.

Decisions

  • Spine stays code-free for OKF (again): tests/okf_sense.rs exercises the bundles through command_exit, no new Rust CheckKind — consistent with M4.
  • Keep the honest negative (single-turn grep competitive) in the live run too; same credibility stance as M4 / TokenMasterX.

EVIDENCE (deterministic, no Copilot spend): 12/12 pytest green; 3/3 spine-sense Rust tests green; both committed bundles + the foreign bundle CONFORMANT (strict); evals/m5-okf-live/ transcript + live-result.json; demo/okf/run-demo.ps1 runs all 5 beats green incl. real red->green via phoenix-mcp. KEY INSIGHT: the OKF arc is closed — Phoenix now produces, gates, senses, heals, consumes, AND interops with open knowledge, enforced in CI. Knowledge is a first-class, self- healing, vendor-neutral artifact, and there's a 3-minute demo to show it.

2026-06-29 — prompt-manifest drift sense ("living prompt document")

Goal: give Phoenix an objective check over its OWN prompt surface — the legit, self-owned analog of a "living document for the system prompt." Capture the 18 skills + AGENTS.md into a content-addressed manifest; SENSE drift (RED, naming the file) vs GREEN-when-matched, through the existing sense/trace/accept spine.

What worked

  • Reused the spine instead of inventing a tool: a fourth CheckKind::PromptManifest (target = baseline manifest path) flows through sense -> trace -> accept for free. No new MCP tool; 3 source files.
  • Failure-first, dogfooded on the real binary: wrote the test first with verify_against stubbed todo!() -> phoenix-mcp sense RED (exit 101) -> implemented the diff -> GREEN -> phoenix-mcp accept ok=true (check_digest 4fa0e55c, trace intact). End-to-end on the real surface: sensing the committed manifest is GREEN; tampering skills/index.md flips RED naming the file.

What didn't work / friction

  • The shipped target/release/phoenix-mcp.exe was stale (built from main) and rejected the new prompt_manifest variant ("unknown variant"). The cargo-test red->green still held (a command_exit check whose subprocess ran the new code), but DIRECT sensing needed cargo build --release. Lesson: a new CheckKind isn't usable via the server until the binary is rebuilt.
  • .phoenix/ is gitignored (runtime: trace, snapshots), so a version-controlled living document can't live there. Moved the committed baseline to docs/prompt-ledger/; workspace_of_manifest falls back to CWD for tracked manifests and uses the .phoenix ancestor only for hermetic tests.

Evidence

  • scripts/ci-local.ps1: ALL GREEN (cargo full suite incl. 2 new prompt_ledger tests; pytest okf 12; phoenix_learn 18; OKF conformance x2). Issue #27; RFC gist linked in the PR.