Dev - #335
Conversation
- Add eval/ directory with 10 pilot skill eval configs - Add GitHub Action (skill-eval.yml) for automated eval on PR - Add generate-eval-config.py script for bootstrapping new evals - Add reusable assertion helpers (skill-quality.js) - Add eval README with setup and usage docs Skills covered: copywriting, cto-advisor, seo-audit, content-strategy, aws-solution-architect, agile-product-owner, senior-frontend, senior-security, mcp-server-builder, launch-strategy CI integration: - Triggers on PR to dev when SKILL.md files change - Detects which skills changed and runs only those evals - Posts results as PR comments (non-blocking) - Uploads full results as artifacts No existing files modified.
Add agents/personas/ directory with: - README.md — what personas are, how to use them, comparison with agents - TEMPLATE.md — format specification for creating new personas - startup-cto.md — technical co-founder persona (engineering + strategy) - growth-marketer.md — bootstrapped growth persona (marketing + analytics) - solo-founder.md — all-in-one indie hacker persona (cross-domain) Each persona includes: - Identity & mindset (personality-driven, not just skill list) - Curated skill loadouts (primary + secondary) - Multi-step workflows for common tasks - Handoff rules for cross-persona collaboration - Anti-patterns to avoid Complements existing agents/ structure — personas for role embodiment, agents for focused task execution.
Restructured all 3 personas + template + README to match the agency-agents agent format: Frontmatter alignment: - name, description, color, emoji, vibe, tools (same fields) - Removed non-standard fields: type, domain, skills, commands, model Body structure alignment: - Second-person voice: 'You are **AgentName**...' - Emoji section headers throughout (🧠 🎯 🚨 📋 🔄 💭 🚀) - Identity & Memory → Core Mission → Critical Rules → Capabilities → Workflows → Communication Style → Success Metrics → Advanced → Learning & Memory - Rich personality backstory (not just skill lists) - Concrete code/workflow examples in each section - Pattern recognition subsection in Learning & Memory Content deepening: - startup-cto: 112 → 196 lines, added due diligence prep, incident response - growth-marketer: 111 → 204 lines, added viral growth, international expansion - solo-founder: 125 → 213 lines, added quit-your-job framework, pivot decisions - Template rewritten to match exact section structure - README updated with correct field documentation
feat: Add persona-based agents (Startup CTO, Growth Marketer, Solo Founder)
🛡️ VirusTotal Security Scan
Scanned with 70+ antivirus engines Automated by ghaction-virustotal |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| --output-format json \ | ||
| 2>&1 | tee /tmp/eval-output.log | ||
|
|
||
| echo "exit_code=$?" >> "$GITHUB_OUTPUT" |
There was a problem hiding this comment.
🟡 $? captures tee's exit code (always 0) instead of npx promptfoo's exit code
On line 121, npx promptfoo ... 2>&1 | tee /tmp/eval-output.log pipes through tee. On line 123, echo "exit_code=$?" captures the exit code of the last command in the pipeline (tee), which almost always succeeds (exit 0), regardless of whether npx promptfoo failed. The intended behavior is clearly to capture promptfoo's exit code, but $? only reflects the rightmost command in a pipeline. The fix is to use ${PIPESTATUS[0]} (bash-specific) or add set -o pipefail at the top of the run: block so that $? reflects the first non-zero exit in the pipeline.
| echo "exit_code=$?" >> "$GITHUB_OUTPUT" | |
| echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT" |
Was this helpful? React with 👍 or 👎 to provide feedback.
|
PR Review by Claude Code CRITICAL: Wrong Target Branch - This PR targets main directly. Per CLAUDE.md, the workflow is feature->dev->main. Please retarget to dev. SECURITY (skill-eval.yml): github.event.inputs.skill and github.base_ref are interpolated directly into shell without sanitization. Fix: pass via env vars (SKILL_INPUT, BASE_REF) and reference those in the shell body instead. SECURITY: In detect-changes, skill_name is interpolated inside a python3 -c string. A name with quotes/newlines could break Python syntax. Pass via os.environ instead. BUG: Pipe exit code - After piping through tee, dollar-sign-? captures tees exit code not promptfoo. Use PIPESTATUS[0] or set -o pipefail. BUG: Redundant RESULTS_FILE fallback re-inlines matrix.skill in Python. Simplify to os.environ[RESULTS_FILE] to fail loudly on misconfiguration. RELIABILITY: promptfoo@latest is unpinned. Pin to a specific version. Also add cache:npm to setup-node to avoid re-downloading on every run. CODE QUALITY: generate-eval-config.py uses raw sys.argv instead of argparse. All scripts must support --help per CLAUDE.md conventions. CODE QUALITY: eval/assertions/skill-quality.js exists but is not referenced by any eval YAML config. Wire it in or document intended usage. DOCS: agents/personas/README.md references ./scripts/convert.sh and ./scripts/install.sh which do not exist. Remove or add them. POSITIVES: Eval YAML configs are well-crafted with specific testable assertions. Personas (startup-cto, growth-marketer, solo-founder) are rich and consistent with agency-agents format. detect-changes only runs evals for changed skills. No existing files modified. |
1 participant
Summary
Checklist
dev(notmain— PRs to main will be auto-closed)SKILL.mdwith valid YAML frontmatter (name,description,license)--helpwithout errorsdomain/skill-name/SKILL.md)Type of Change
Testing