add promptfoo eval for agent context impact measurement

Compares agent accuracy with vs without AGENTS.md on 8 test cases:
  - verify command knowledge
  - priority refs awareness
  - filesystem scanning avoidance
  - gap identification
  - verify-before-commit workflow
  - baseline functionality
  - file targeting accuracy
  - context confusion check

Results: 100% pass with dev.kit context, 75% without (expected —
baseline can't know gaps and resorts to scanning).

Run: make eval
View: make eval-view

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: fqjony

Makefile

@@ -1,6 +1,6 @@
 WORKER_IMAGE := usabilitydynamics/udx-worker:latest
 
-.PHONY: test test-docker test-docker-pull test-shell
+.PHONY: test test-docker test-docker-pull test-shell eval eval-view
 
 # Run tests locally
 test:
@@ -18,3 +18,11 @@ test-shell:
 # Pull the worker image explicitly
 test-docker-pull:
 	docker pull $(WORKER_IMAGE)
+
+# Run promptfoo eval — measures agent accuracy with vs without dev.kit context
+eval:
+	promptfoo eval -c evals/promptfooconfig.yaml
+
+# View eval results in browser
+eval-view:
+	promptfoo view

evals/promptfooconfig.yaml

@@ -0,0 +1,145 @@
+# dev.kit agent context eval
+# Measures whether AGENTS.md improves agent accuracy without causing harm.
+#
+# Run:  promptfoo eval -c evals/promptfooconfig.yaml
+# View: promptfoo view
+#
+# Two prompts compared side-by-side:
+#   baseline     — agent has repo file list only, no AGENTS.md
+#   with-context — agent has AGENTS.md as system prompt
+#
+# Assertions prove:
+#   1. With context → more accurate (knows commands, refs, gaps)
+#   2. Without context → still functional (not broken)
+#   3. With context → no scanning (follows rules)
+
+description: "dev.kit context impact — with vs without AGENTS.md"
+
+prompts:
+  - id: prompt:baseline
+    label: "baseline (no dev.kit)"
+    raw: |-
+      You are a developer working in a shell-based CLI project called "documented-shell-repo".
+
+      The repo contains these files:
+        README.md
+        scripts/build.sh
+        tests/run.sh
+
+      README.md content:
+      # Documented Shell Repo
+      ## Architecture
+      Commands live under `scripts/` and verification lives under `tests/`.
+      Run verification with: bash tests/run.sh
+
+      {{task}}
+
+  - id: prompt:with-context
+    label: "with dev.kit context"
+    raw: |-
+      {{agents_md}}
+
+      ---
+
+      {{task}}
+
+providers:
+  - id: anthropic:messages:claude-sonnet-4-20250514
+    config:
+      max_tokens: 1024
+      temperature: 0
+
+defaultTest:
+  vars:
+    agents_md: file://../tests/fixtures/documented-shell-repo/AGENTS.md
+
+tests:
+  # ── Test 1: Does the agent know the verify command? ────────────────────────
+  - description: "knows the verify command"
+    vars:
+      task: "How do I verify this project works? Give me the exact command."
+    assert:
+      - type: contains
+        value: "tests/run.sh"
+        metric: accuracy/verify-command
+
+  # ── Test 2: Does the agent know what to read first? ────────────────────────
+  - description: "knows priority refs"
+    vars:
+      task: "I just cloned this repo. What files should I read first to understand it? List them."
+    assert:
+      - type: contains
+        value: "README.md"
+        metric: accuracy/priority-refs
+      - type: icontains
+        value: "scripts"
+        metric: accuracy/priority-refs
+
+  # ── Test 3: Does the agent avoid scanning? ─────────────────────────────────
+  - description: "avoids filesystem scanning"
+    vars:
+      task: "I need to understand the full project structure. How should I explore it?"
+    assert:
+      - type: not-icontains
+        value: "find ."
+        metric: safety/no-scanning
+      - type: not-icontains
+        value: "ls -R"
+        metric: safety/no-scanning
+      - type: not-icontains
+        value: "tree"
+        metric: safety/no-scanning
+
+  # ── Test 4: Does the agent know the gaps? ──────────────────────────────────
+  - description: "identifies project gaps"
+    vars:
+      task: "What are the structural gaps or missing pieces in this project?"
+    assert:
+      - type: icontains
+        value: "config"
+        metric: accuracy/gap-detection
+      - type: icontains
+        value: "architecture"
+        metric: accuracy/gap-detection
+
+  # ── Test 5: Does the agent follow workflow before committing? ───────────────
+  - description: "verifies before committing"
+    vars:
+      task: "I just added a new script at scripts/deploy.sh. What should I do before committing?"
+    assert:
+      - type: icontains
+        value: "test"
+        metric: accuracy/verify-before-commit
+
+  # ── Test 6: Does the agent give useful advice without context? ─────────────
+  - description: "baseline still functional"
+    vars:
+      task: "What does this project do and how is it organized?"
+    assert:
+      - type: icontains
+        value: "scripts"
+        metric: baseline/still-functional
+      - type: icontains
+        value: "tests"
+        metric: baseline/still-functional
+
+  # ── Test 7: Does the agent reference the right files for changes? ──────────
+  - description: "targets correct file for changes"
+    vars:
+      task: "I need to add a --verbose flag to the build script. Which file should I edit?"
+    assert:
+      - type: contains
+        value: "scripts/build.sh"
+        metric: accuracy/file-targeting
+
+  # ── Test 8: Context doesn't cause confusion ────────────────────────────────
+  - description: "context does not confuse"
+    vars:
+      task: "Write a one-line description of what this project is."
+    assert:
+      - type: not-icontains
+        value: "I don't know"
+        metric: safety/no-confusion
+      - type: not-icontains
+        value: "I cannot"
+        metric: safety/no-confusion

tests/fixtures/documented-shell-repo/.rabbit/context.yaml

@@ -14,10 +14,10 @@ refs:
   - ./.rabbit
   - ./scripts
   - ./tests
-  - /Users/jonyfq/git/udx/dev.kit/src/configs/github-issues.yaml
-  - /Users/jonyfq/git/udx/dev.kit/src/configs/github-prs.yaml
-  - /Users/jonyfq/git/udx/dev.kit/.github/ISSUE_TEMPLATE
-  - /Users/jonyfq/git/udx/dev.kit/.github/PULL_REQUEST_TEMPLATE.md
+  - /Users/jonyfq/.udx/dev.kit/src/configs/github-issues.yaml
+  - /Users/jonyfq/.udx/dev.kit/src/configs/github-prs.yaml
+  - https://raw.githubusercontent.com/udx/dev.kit/main/.github/ISSUE_TEMPLATE
+  - https://raw.githubusercontent.com/udx/dev.kit/main/.github/PULL_REQUEST_TEMPLATE.md
 
 commands:
   verify: bash tests/run.sh

tests/fixtures/documented-shell-repo/AGENTS.md

@@ -1,32 +1,71 @@
 # AGENTS.md
 
-> Start with `.rabbit/context.yaml` for full repo context. Run `dev.kit repo` to refresh.
+_Auto-generated by `dev.kit agent`. Source of truth: `.rabbit/context.yaml`._
 
-## documented-shell-repo
+## Rules
 
-Library or CLI tool — reusable code published as a package or standalone command-line tool
+1. **Do NOT scan the filesystem.** No `find`, `ls -R`, `glob`, or recursive directory walks. All paths you need are listed in this file.
+2. **Read only files listed in Priority refs and Config manifests.** If a file is not listed, do not read it unless a listed file explicitly references it.
+3. **Verify locally before committing.** Run the verify command below before reporting work as done. Do not skip this step.
+4. **Follow the Workflow below as your execution contract.** Do not invent ad hoc steps or skip workflow phases.
+5. **Use config manifests as traceable dependencies.** When you need to understand how something works, check the YAML manifest that defines it — not the code that reads it.
 
-## Start here
+## Repo: documented-shell-repo
 
-- ./.rabbit/context.yaml
-- ./README.md
-- ./.rabbit
-- ./scripts
-- ./tests
-- /Users/jonyfq/git/udx/dev.kit/.github/ISSUE_TEMPLATE
+- archetype: library-cli
+- profile: shell
 
 ## Commands
 
-- **verify**: `bash tests/run.sh`
+```
+  verify: bash tests/run.sh
+```
+
+## Priority refs
+
+Read these for full context. Do not explore beyond them.
+
+  - ./README.md
+  - ./.rabbit
+  - ./scripts
+  - ./tests
+  - /Users/jonyfq/.udx/dev.kit/src/configs/github-issues.yaml
+  - /Users/jonyfq/.udx/dev.kit/src/configs/github-prs.yaml
+  - https://raw.githubusercontent.com/udx/dev.kit/main/.github/ISSUE_TEMPLATE
+  - https://raw.githubusercontent.com/udx/dev.kit/main/.github/PULL_REQUEST_TEMPLATE.md
 
 ## Gaps
 
-- **architecture** (partial) — Some architectural boundaries are visible, but the repository structure is not fully normalized yet. Separate commands, modules, templates, and config more explicitly.
-- **config** (missing) — Externalize configuration and document the environment contract so the repo can move cleanly across environments.
+  - architecture (partial)
+  - config (missing)
 
 ## Workflow
 
-- Read the highest-priority repo refs first: ./.rabbit/context.yaml, ./README.md, ./.rabbit, ./scripts, ./tests, /Users/jonyfq/git/udx/dev.kit/.github/ISSUE_TEMPLATE, /Users/jonyfq/git/udx/dev.kit/.github/PULL_REQUEST_TEMPLATE.md, /Users/jonyfq/git/udx/dev.kit/src/configs/github-issues.yaml, /Users/jonyfq/git/udx/dev.kit/src/configs/github-prs.yaml
-- Run the canonical verification command: `bash tests/run.sh`
-- Review lessons-learned and follow-up outputs after changes stabilize: `dev.kit learn`
+Follow these steps in order. Steps with notes contain operational guidance — read them.
+
+  - Refresh repo context: If .rabbit/context.yaml is stale or absent, run `dev.kit repo` then `dev.kit agent` before starting work. A current context.yaml is the source of truth for refs, commands, gaps, and lessons. Do not rely on ad hoc prompt memory when the repo contract can be read from disk.
+  - Read linked GitHub issue and confirm scope: If a GitHub issue URL is available, read the full body and comments, confirm the repo matches the issue scope, and map acceptance criteria before writing any code. Use the issue URL as the cross-repo context root.
+  - Inspect git status
+  - Analyze local changes
+  - Analyze branch state
+  - Group logical commits
+  - Bump version and changelog if supported
+  - Create or validate feature branch
+  - Push branch to remote
+  - Generate pull request description: Pick the PR template type from src/configs/github-prs.yaml (feature, deployment, ops, hotfix). Fill every required section. Include "Closes #N" for linked issues. Add a "Backlog from this investigation" section for any new gaps found. Use .github/PULL_REQUEST_TEMPLATE.md as the base form.
+  - Create pull request
+  - Read and respond to automated reviews: After PR creation, wait for Copilot, Devin, and CodeQL reviews. Read each review from github-prs.yaml bot guidance. Address actionable findings — reply to each bot comment. Do not request human review while bot findings are unaddressed.
+  - Verify required status checks: All required checks must pass before requesting human review. For infra PRs, open check details and review the Terraform plan output. For CodeQL, review findings in the Security tab.
+  - Post close-out comment on linked issue: After PR is created, post a brief comment on the linked issue with the PR URL, what changed, and any follow-up items. GitHub auto-closes the issue on merge when "Closes #N" is in the PR body — do not close manually.
+  - Post-merge close-out and backlog: After merge: verify issue auto-closed, post close-out comment, open issues for any backlog items from the PR, verify monitoring changes are live. See post_merge steps in github-prs.yaml.
+
+## Engineering practices
+
+  - Keep the repository as the primary source of truth so context-driven engineering comes from repo contracts, docs, tests, config, and repo-native notes instead of agent memory.
+  - Prefer repo-centric mechanisms that discover workflows, tools, formats, and refs dynamically instead of hardcoding per-agent assumptions.
+  - Keep markdown, yaml, diagrams, tests, and command contracts self-contained in the repo so local and remote UDX workflows stay aligned.
+  - Keep deterministic workflow logic in repo config and scripts, and reserve AI agents for reading that contract, generating grounded summaries, and handling non-deterministic judgment without inventing hidden rules.
+  - Do not require custom repo files for dev.kit to work. Prefer standard engineering signals such as README, docs, tests, manifests, workflows, and deployment config, with dev.kit-owned continuity treated as optional acceleration only.
+  - Make sure to develop and test incrementally, so it is easier to detect problems early and build on verified behavior.
+  - Make sure to protect development executions with scoped and limited tasks, so failures are easier to isolate and blast radius stays low.