feat: autoresearch-agent — bug fixes, plugin packaging, 5 slash commands by alirezarezvani · Pull Request #349 · alirezarezvani/claude-skills

alirezarezvani · 2026-03-13T13:39:28Z

Summary

Fix critical revert bug in run_experiment.py — was saving HEAD as pre_commit (no-op revert when agent already committed), now uses git reset --hard HEAD~1
Remove broken --loop mode — the AI agent IS the loop; the script handles one iteration via --single
Fix shell injection — all git commands use subprocess.run(["git", ...]) instead of shell=True
Fix 6 falsy-zero bugs in log_results.py where baseline=0.0 was treated as None
Package as Claude Code plugin with 5 slash commands (/ar:setup, /ar:run, /ar:loop, /ar:status, /ar:resume)
Configurable loop intervals — user selects 10m, 1h, daily, weekly, or monthly via /ar:loop
Cross-platform sync — Codex CLI (157), Gemini CLI (229), convert.sh (all 7 tools) confirmed
6 new MkDocs pages + marketplace plugin docs(workflow): establish branch protection and fix CI quality gate #20 registered

Changed Files (39)

Core fixes (10 files):

scripts/run_experiment.py — revert fix, remove --loop, fix shell injection, add --description
scripts/setup_experiment.py — fix shell injection, remove dead flag, fix docstring parsing
scripts/log_results.py — fix falsy-zero, add domain_filter, move imports
5 evaluators — FileNotFoundError handling, output fixes, unused imports
references/program-template.md — fix GPT-4o refs, scoring formula

Plugin packaging (9 new files):

.claude-plugin/plugin.json, settings.json, CLAUDE.md
skills/{setup,run,loop,status,resume}/SKILL.md
agents/experiment-runner.md

SKILL.md rewrite:

Clear "Agent Protocol" replacing ambiguous "Loop Protocol"
Slash commands table, results.tsv format spec, strategy escalation

Docs & infra (20 files):

marketplace.json, CLAUDE.md, README.md, mkdocs.yml, docs/index.md
6 new MkDocs skill pages, Gemini symlinks, skills-index updates

Test plan

All 3 scripts pass python3 script.py --help
All 8 evaluators pass python3 -c "import ast; ast.parse(...)"
E2E test: setup → dry-run → single (KEEP) → single (DISCARD with revert) → dashboard
Revert verified: file content, git log, results.tsv all correct after DISCARD
No shell=True on git commands (only on user eval commands, intentional)
No --loop/--resume refs in SKILL.md
MkDocs build passes (228 pages, 6.96s)
Count consistency: 17 agents, 22 commands across all docs
All 20 marketplace plugin paths validated

🤖 Generated with Claude Code

… slash commands **Bug fixes (run_experiment.py):** - Fix broken revert logic: was saving HEAD as pre_commit (no-op revert), now uses git reset --hard HEAD~1 for correct rollback - Remove broken --loop mode (agent IS the loop, script handles one iteration) - Fix shell injection: all git commands use subprocess list form - Replace shell tail with Python file read **Bug fixes (other scripts):** - setup_experiment.py: fix shell injection in git branch creation, remove dead --skip-baseline flag, fix evaluator docstring parsing - log_results.py: fix 6 falsy-zero bugs (baseline=0 treated as None), add domain_filter to CSV/markdown export, move import time to top - evaluators: add FileNotFoundError handling, fix output format mismatch in llm_judge_copy, add peak_kb on macOS, add ValueError handling **Plugin packaging (NEW):** - plugin.json, settings.json, CLAUDE.md for plugin registry - 5 slash commands: /ar:setup, /ar:run, /ar:loop, /ar:status, /ar:resume - /ar:loop supports user-selected intervals (10m, 1h, daily, weekly, monthly) - experiment-runner agent for autonomous loop iterations - Registered in marketplace.json as plugin #20 **SKILL.md rewrite:** - Replace ambiguous "Loop Protocol" with clear "Agent Protocol" - Add results.tsv format spec, strategy escalation, self-improvement - Replace "NEVER STOP" with resumable stopping logic **Docs & sync:** - Codex (157 skills), Gemini (229 items), convert.sh all pick up the skill - 6 new MkDocs pages, mkdocs.yml nav updated - Counts updated: 17 agents, 22 slash commands Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-03-13T13:39:51Z

🔒 Skill Security Audit Results

❌ `engineering` — FAIL

Severity	Count
🔴 Critical	29
🟡 High	12

Findings detail

{
  "severity": "HIGH",
  "category": "FS-HIDDEN",
  "file": ".claude-plugin",
  "line": 0,
  "pattern": ".claude-plugin",
  "risk": "Hidden file '.claude-plugin' \u2014 may contain secrets or hidden config",
  "fix": "Remove hidden files from skill distribution"
}
{
  "severity": "HIGH",
  "category": "FS-HIDDEN",
  "file": ".codex",
  "line": 0,
  "pattern": ".codex",
  "risk": "Hidden file '.codex' \u2014 may contain secrets or hidden config",
  "fix": "Remove hidden files from skill distribution"
}
{
  "severity": "HIGH",
  "category": "FS-HIDDEN",
  "file": "autoresearch-agent/.claude-plugin",
  "line": 0,
  "pattern": ".claude-plugin",
  "risk": "Hidden file '.claude-plugin' \u2014 may contain secrets or hidden config",
  "fix": "Remove hidden files from skill distribution"
}
{
  "severity": "CRITICAL",
  "category": "NET-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/tech-debt-tracker/assets/sample_codebase/src/payment_processor.py",
  "line": 100,
  "pattern": "response = requests.post(",
  "risk": "Outbound HTTP write request \u2014 potential data exfiltration",
  "fix": "Remove outbound POST/PUT/PATCH or verify destination is trusted and necessary"
}
{
  "severity": "CRITICAL",
  "category": "NET-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/tech-debt-tracker/assets/sample_codebase/src/payment_processor.py",
  "line": 142,
  "pattern": "response = requests.post(",
  "risk": "Outbound HTTP write request \u2014 potential data exfiltration",
  "fix": "Remove outbound POST/PUT/PATCH or verify destination is trusted and necessary"
}
{
  "severity": "CRITICAL",
  "category": "NET-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/tech-debt-tracker/assets/sample_codebase/src/payment_processor.py",
  "line": 182,
  "pattern": "response = requests.post(",
  "risk": "Outbound HTTP write request \u2014 potential data exfiltration",
  "fix": "Remove outbound POST/PUT/PATCH or verify destination is trusted and necessary"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_size.py",
  "line": 26,
  "pattern": "result = subprocess.run(BUILD_CMD, shell=True, capture_output=True)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_size.py",
  "line": 34,
  "pattern": "subprocess.run(DOCKER_BUILD_CMD, shell=True, capture_output=True)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/test_pass_rate.py",
  "line": 13,
  "pattern": "result = subprocess.run(TEST_CMD, shell=True, capture_output=True, text=True, timeout=300)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/build_speed.py",
  "line": 20,
  "pattern": "subprocess.run(CLEAN_CMD, shell=True, capture_output=True, timeout=60)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/build_speed.py",
  "line": 23,
  "pattern": "result = subprocess.run(BUILD_CMD, shell=True, capture_output=True, timeout=600)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_speed.py",
  "line": 20,
  "pattern": "subprocess.run(COMMAND, shell=True, capture_output=True, timeout=120)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_speed.py",
  "line": 25,
  "pattern": "result = subprocess.run(COMMAND, shell=True, capture_output=True, timeout=120)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 125,
  "pattern": "\"risk\": \"Arbitrary command execution via os.system()\",",
  "risk": "Arbitrary command execution via os.system()",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 132,
  "pattern": "\"risk\": \"Command execution via os.popen()\",",
  "risk": "Command execution via os.popen()",
  "fix": "Use subprocess.run() with list arguments and capture_output=True"
}
{
  "severity": "CRITICAL",
  "category": "CODE-EXEC",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 154,
  "pattern": "\"risk\": \"Arbitrary code execution via eval()\",",
  "risk": "Arbitrary code execution via eval()",
  "fix": "Use ast.literal_eval() for data parsing or explicit parsing logic"
}
{
  "severity": "CRITICAL",
  "category": "CODE-EXEC",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 161,
  "pattern": "\"risk\": \"Arbitrary code execution via exec()\",",
  "risk": "Arbitrary code execution via exec()",
  "fix": "Remove exec() \u2014 rewrite logic to avoid dynamic code execution"
}
{
  "severity": "CRITICAL",
  "category": "CODE-EXEC",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 162,
  "pattern": "\"fix\": \"Remove exec() \u2014 rewrite logic to avoid dynamic code execution\",",
  "risk": "Arbitrary code execution via exec()",
  "fix": "Remove exec() \u2014 rewrite logic to avoid dynamic code execution"
}
{
  "severity": "CRITICAL",
  "category": "PRIV-ESC",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 342,
  "pattern": "\"fix\": \"Remove sudo usage. Skills should never require elevated privileges\",",
  "risk": "Sudo invocation \u2014 privilege escalation attempt",
  "fix": "Remove sudo usage. Skills should never require elevated privileges"
}
{
  "severity": "HIGH",
  "category": "DESERIAL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 378,
  "pattern": "\"fix\": \"Use yaml.safe_load() or yaml.load(data, Loader=yaml.SafeLoader)\",",
  "risk": "Unsafe YAML loading \u2014 can execute arbitrary code",
  "fix": "Use yaml.safe_load() or yaml.load(data, Loader=yaml.SafeLoader)"
}
{
  "severity": "CRITICAL",
  "category": "PRIV-ESC",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 468,
  "pattern": "\"regex\": r\"<!--\\s*(?:system|instruction|override|ignore|execute|run|sudo|admin)\",",
  "risk": "Sudo invocation \u2014 privilege escalation attempt",
  "fix": "Remove sudo usage. Skills should never require elevated privileges"
}
{
  "severity": "CRITICAL",
  "category": "CODE-EXEC",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 579,
  "pattern": "\"risk\": \"Dynamic Function constructor \u2014 equivalent to eval()\",",
  "risk": "Arbitrary code execution via eval()",
  "fix": "Use ast.literal_eval() for data parsing or explicit parsing logic"
}
{
  "severity": "HIGH",
  "category": "FS-ABUSE",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 917,
  "pattern": "shutil.rmtree(tmp_dir, ignore_errors=True)",
  "risk": "Recursive directory deletion \u2014 destructive operation",
  "fix": "Remove or restrict to specific, validated paths within skill scope"
}
{
  "severity": "HIGH",
  "category": "FS-ABUSE",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 929,
  "pattern": "shutil.rmtree(tmp_dir, ignore_errors=True)",
  "risk": "Recursive directory deletion \u2014 destructive operation",
  "fix": "Remove or restrict to specific, validated paths within skill scope"
}
{
  "severity": "HIGH",
  "category": "FS-ABUSE",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 1045,
  "pattern": "shutil.rmtree(cleanup_dir, ignore_errors=True)",
  "risk": "Recursive directory deletion \u2014 destructive operation",
  "fix": "Remove or restrict to specific, validated paths within skill scope"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/env-secrets-manager/references/secret-patterns.md",
  "line": 8,
  "pattern": "- GitHub personal access tokens (`ghp_...`)",
  "risk": "Instruction to access sensitive files or credentials",
  "fix": "Remove credential/sensitive file access directives"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/env-secrets-manager/references/validation-detection-rotation.md",
  "line": 304,
  "pattern": "- **GitHub PAT**: Settings \u2192 Developer Settings \u2192 Personal access tokens \u2192 Revoke \u2192 Create new",
  "risk": "Instruction to access sensitive files or credentials",
  "fix": "Remove credential/sensitive file access directives"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-OVERRIDE",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/SKILL.md",
  "line": 60,
  "pattern": "| **System prompt override** | \"Ignore previous instructions\", \"You are now...\" | \ud83d\udd34 CRITICAL |",
  "risk": "Attempts to override system prompt and prior instructions",
  "fix": "Remove instruction override attempts"
}
{
  "severity": "CRITICAL",
  "category": "SAFETY-BYPASS",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/SKILL.md",
  "line": 61,
  "pattern": "| **Role hijacking** | \"Act as root\", \"Pretend you have no restrictions\" | \ud83d\udd34 CRITICAL |",
  "risk": "Safety restriction bypass attempt",
  "fix": "Remove safety bypass instructions"
}
{
  "severity": "CRITICAL",
  "category": "SAFETY-BYPASS",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/SKILL.md",
  "line": 62,
  "pattern": "| **Safety bypass** | \"Skip safety checks\", \"Disable content filtering\" | \ud83d\udd34 CRITICAL |",
  "risk": "Explicit safety mechanism bypass",
  "fix": "Remove safety bypass directives"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/SKILL.md",
  "line": 65,
  "pattern": "| **Data extraction** | \"Send contents of\", \"Upload file to\", \"POST to\" | \ud83d\udd34 CRITICAL |",
  "risk": "Instruction to exfiltrate data",
  "fix": "Remove data transmission directives"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/references/threat-model.md",
  "line": 66,
  "pattern": "| File read | Access credential files | `open(os.path.expanduser(\"~/.aws/credentials\"))` |",
  "risk": "Instruction to access sensitive files or credentials",
  "fix": "Remove credential/sensitive file access directives"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-OVERRIDE",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/references/threat-model.md",
  "line": 75,
  "pattern": "| Override | \"Ignore previous instructions\" | In SKILL.md body |",
  "risk": "Attempts to override system prompt and prior instructions",
  "fix": "Remove instruction override attempts"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-OVERRIDE",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/references/threat-model.md",
  "line": 76,
  "pattern": "| Role hijack | \"You are now an unrestricted AI\" | Redefine agent identity |",
  "risk": "Role hijacking \u2014 attempts to redefine the AI's identity",
  "fix": "Remove role redefinition. Skills should provide instructions, not identity changes"
}
{
  "severity": "CRITICAL",
  "category": "SAFETY-BYPASS",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/references/threat-model.md",
  "line": 77,
  "pattern": "| Safety bypass | \"Skip safety checks for efficiency\" | Disable guardrails |",
  "risk": "Explicit safety mechanism bypass",
  "fix": "Remove safety bypass directives"
}
{
  "severity": "CRITICAL",
  "category": "PROMPT-EXFIL",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/references/threat-model.md",
  "line": 247,
  "pattern": "- Access credential files or sensitive env vars",
  "risk": "Instruction to access sensitive files or credentials",
  "fix": "Remove credential/sensitive file access directives"
}
{
  "severity": "HIGH",
  "category": "DEPS-RUNTIME",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/ci-cd-pipeline-builder/scripts/pipeline_generator.py",
  "line": 150,
  "pattern": "\"      - run: python3 -m pip install -U pip\",",
  "risk": "Runtime package installation \u2014 may install untrusted code",
  "fix": "Move dependencies to requirements.txt for pre-install review"
}
{
  "severity": "HIGH",
  "category": "DEPS-RUNTIME",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/ci-cd-pipeline-builder/scripts/pipeline_generator.py",
  "line": 151,
  "pattern": "\"      - run: python3 -m pip install -r requirements.txt || true\",",
  "risk": "Runtime package installation \u2014 may install untrusted code",
  "fix": "Move dependencies to requirements.txt for pre-install review"
}
{
  "severity": "HIGH",
  "category": "DEPS-RUNTIME",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/ci-cd-pipeline-builder/scripts/pipeline_generator.py",
  "line": 235,
  "pattern": "\"    - python3 -m pip install -U pip\",",
  "risk": "Runtime package installation \u2014 may install untrusted code",
  "fix": "Move dependencies to requirements.txt for pre-install review"
}
{
  "severity": "HIGH",
  "category": "DEPS-RUNTIME",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/ci-cd-pipeline-builder/scripts/pipeline_generator.py",
  "line": 236,
  "pattern": "\"    - python3 -m pip install -r requirements.txt || true\",",
  "risk": "Runtime package installation \u2014 may install untrusted code",
  "fix": "Move dependencies to requirements.txt for pre-install review"
}
{
  "severity": "HIGH",
  "category": "DEPS-RUNTIME",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/skill-security-auditor/scripts/skill_security_auditor.py",
  "line": 717,
  "pattern": "# Check for pip/npm install in code",
  "risk": "Runtime package installation \u2014 may install untrusted code",
  "fix": "Move dependencies to package.json for pre-install review"
}

❌ `engineering/autoresearch-agent` — FAIL

Severity	Count
🔴 Critical	7
🟡 High	1

Findings detail

{
  "severity": "HIGH",
  "category": "FS-HIDDEN",
  "file": ".claude-plugin",
  "line": 0,
  "pattern": ".claude-plugin",
  "risk": "Hidden file '.claude-plugin' \u2014 may contain secrets or hidden config",
  "fix": "Remove hidden files from skill distribution"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_size.py",
  "line": 26,
  "pattern": "result = subprocess.run(BUILD_CMD, shell=True, capture_output=True)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_size.py",
  "line": 34,
  "pattern": "subprocess.run(DOCKER_BUILD_CMD, shell=True, capture_output=True)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/test_pass_rate.py",
  "line": 13,
  "pattern": "result = subprocess.run(TEST_CMD, shell=True, capture_output=True, text=True, timeout=300)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/build_speed.py",
  "line": 20,
  "pattern": "subprocess.run(CLEAN_CMD, shell=True, capture_output=True, timeout=60)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/build_speed.py",
  "line": 23,
  "pattern": "result = subprocess.run(BUILD_CMD, shell=True, capture_output=True, timeout=600)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_speed.py",
  "line": 20,
  "pattern": "subprocess.run(COMMAND, shell=True, capture_output=True, timeout=120)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}
{
  "severity": "CRITICAL",
  "category": "CMD-INJECT",
  "file": "/home/runner/work/claude-skills/claude-skills/engineering/autoresearch-agent/evaluators/benchmark_speed.py",
  "line": 25,
  "pattern": "result = subprocess.run(COMMAND, shell=True, capture_output=True, timeout=120)",
  "risk": "Shell injection via subprocess with shell=True",
  "fix": "Use subprocess.run() with list arguments and shell=False"
}

claude · 2026-03-13T13:45:19Z

PR Review: feat: autoresearch-agent — bug fixes, plugin packaging, 5 slash commands

Overall this is a solid, well-scoped PR with meaningful security and correctness fixes.

What is Well Done

Security fix (shell injection): The refactor from shell=True git commands to list-based subprocess calls in both run_experiment.py and setup_experiment.py is the right approach. The comment documenting the intentional shell=True on eval_cmd (user-provided, may contain pipes/redirects) is clear and honest.

Critical revert bug fix: The change from git reset --hard {pre_commit} (where pre_commit was HEAD at runtime, effectively a no-op) to git reset --hard HEAD~1 is correct. The original bug meant failed experiments were never truly discarded.

Falsy-zero fixes in log_results.py: All six if baseline and best to if baseline is not None and best is not None fixes are correct and consistent. Ditto for the best_str/pct_str formatting guards.

import time moved to top-level: Correct cleanup — import time was inside a conditional block before.

Bugs and Issues

1. git reset --hard HEAD~1 on an initial commit will silently fail

In run_single(), if the experiment is the first commit in the repo (no parent), HEAD~1 does not exist and the reset will fail with a non-zero exit code. The current code ignores the return value of all run_git(["reset", ...]) calls — a failed revert would leave the repo in a dirty state with no error surfaced to the user. Consider logging a warning if the return code is non-zero.

2. --resume flag removed without cleanup

get_last_active() was deleted, but the PR summary only mentions removing --loop. Worth confirming the --resume argparse definition (if it still exists) and all SKILL.md references were also cleaned up.

3. llm_judge_copy.py fallback regex is overly broad

The fallback parsing block matches any "word: number" line — including things like "timeout: 30" or arbitrary LLM annotations. A 5-item cap or checking against a known criterion set would reduce false positives. Also, import re is still inside the if not found_scores: block — it should be at the top of the file with the other imports.

4. settings.json is a non-standard file

The new engineering/autoresearch-agent/settings.json uses fields (platforms, category, tags, commands, agents) not in the CLAUDE.md plugin.json schema. The .claude-plugin/plugin.json is correctly formed, but settings.json could cause confusion about which file is authoritative. Consider documenting it as an internal/development manifest or removing it.

Minor Issues

5. Template/placeholder artifacts in docs and skills index

docs/agents/index.md adds cards for "Persona-Based Agents" (readme.md) and "Agent Name Agent Personality" (template.md) — these look like boilerplate that leaked into the published docs index. Similarly, .gemini/skills-index.json adds entries with descriptions like "One paragraph describing what this agent does, who it's for, and when to activate it." — placeholder text from the agent template. These should not appear in production indexes.

6. Agent count discrepancy: docs say 21, README/CLAUDE.md say 17

docs/agents/index.md was updated to say "21 agents" (includes 5 new persona/template cards), while README.md, CLAUDE.md, and marketplace.json all say "17 agents". The .gemini/skills-index.json category total also shows count 21. Pick a canonical count and keep it consistent across all files.

7. Removed --skip-baseline without noting it as a breaking change

setup_experiment.py drops the --skip-baseline argparse flag. This is a breaking change for any automation or docs that reference it and is not mentioned in the PR summary.

8. No enforcement of single loop per experiment

The SKILL.md for /ar:loop says "Don't start multiple loops for the same experiment" but there is no guard: a second /ar:loop would overwrite loop.json, orphaning the first cron job (it would keep running with no record of its ID). A simple check — read loop.json and error out if cron_id is already set — would prevent this.

Suggestions (non-blocking)

get_description_from_diff: When called on the first experiment run, HEAD~1 may not exist. It gracefully falls back to "experiment" already, but checking exp_num == 1 and skipping the diff call would be cleaner.
memory_usage.py: Adding peak_kb alongside peak_mb is useful, but existing experiment configs using metric_grep for peak_mb will not automatically pick up the new field.
Stale loop detection: Consider having /ar:status surface a warning if loop.json exists but the cron job is older than 3 days.

Summary

Category	Finding	Severity
Bug	run_git return value ignored on revert failure	Low
Bug	Broad fallback regex in llm_judge_copy.py and import re inline	Low
Inconsistency	Agent count mismatch (17 vs 21) across files	Medium
Artifact	Template/placeholder entries in docs and skills-index	Medium
Breaking Change	--skip-baseline removed without changelog note	Low
Design	No enforcement of single-loop-per-experiment	Low
Non-standard	settings.json uses undocumented schema fields	Low

The core fixes (revert bug, shell injection, falsy-zero) are all correct. Addressing the agent count inconsistency and removing the template artifacts from the docs index would be good cleanup before merging.

Generated with Claude Code

alirezarezvani merged commit 536c3df into dev Mar 13, 2026
4 of 5 checks passed

alirezarezvani deleted the feat/autoresearch-plugin branch March 13, 2026 13:41

alirezarezvani mentioned this pull request Mar 13, 2026

Release: autoresearch v2.0 + Custom GPTs cleanup + plugin packaging #354

Merged

Navigation Menu

Uh oh!

feat: autoresearch-agent — bug fixes, plugin packaging, 5 slash commands - #349

feat: autoresearch-agent — bug fixes, plugin packaging, 5 slash commands#349
alirezarezvani merged 1 commit into
devfrom
feat/autoresearch-plugin

alirezarezvani commented Mar 13, 2026

Uh oh!

github-actions Bot commented Mar 13, 2026

Uh oh!

Uh oh!

claude Bot commented Mar 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

alirezarezvani commented Mar 13, 2026

Summary

Changed Files (39)

Test plan

Uh oh!

github-actions Bot commented Mar 13, 2026

🔒 Skill Security Audit Results

❌ engineering — FAIL

❌ engineering/autoresearch-agent — FAIL

Uh oh!

Uh oh!

claude Bot commented Mar 13, 2026

PR Review: feat: autoresearch-agent — bug fixes, plugin packaging, 5 slash commands

What is Well Done

Bugs and Issues

Minor Issues

Suggestions (non-blocking)

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

❌ `engineering` — FAIL

❌ `engineering/autoresearch-agent` — FAIL