Six reliability/UX gaps hit during a real-world /graphify --update run (incl. a PII risk)

[smoshe6]

Summary

Ran /graphify update on a real production monorepo (existing .graphifyignore scoping the graph to product source, 12k+ pre-existing nodes). Hit six concrete rough edges — one a genuine data-handling risk (PII), the rest reliability/UX gaps in the incremental-update and labeling flow. Details and repro below.

1. No signal before extracting sensitive files (PII risk)

--update detected ~130 changed/new files, including real PDF resumes/CVs that had landed in the working tree (containing a real person's name, contact info, work history). Nothing in detect_incremental or the semantic-extraction dispatch distinguishes "a person's resume" from "a design doc" — both are just file_type: paper/document and get queued for the same subagent pipeline, which would read the PDF and extract named entities (i.e. that person's identity and contact info) straight into graph.json.

Suggestion: a cheap heuristic gate before dispatch — filename patterns (*cv*, *resume*), or for PDFs a quick "does page 1 look like a single person's contact card" check — that flags matches for confirmation instead of silently extracting. Right now the entire burden is on the operator to notice and hand-edit .graphifyignore before every run; that's a trap for anyone less careful about PII than a given project's own conventions demand.

2. .graphifyignore patterns silently no-op on quoted/space-containing paths

Wrote a line like this to exclude a directory whose name contains a space:

"some directory name/"

It matched nothing — no error, no warning. All files under that directory were still picked up as new on the next detect_incremental run. Removing the quotes fixed it immediately.

Suggestion: either support quoted paths (gitignore syntax doesn't use quotes, so intuitively people coming from other tools will try it anyway), or validate every .graphifyignore pattern against the corpus at load time and warn on patterns with zero matches ("dead ignore rule: line N matched 0 files"). A config file whose only job is exclusion should never fail silently.

3. Community re-labeling after cluster-only is a three-tool relay with visible seams

This project labels communities via its own local script (highest-degree-member naming, no LLM cost) instead of graphify label, specifically to avoid LLM spend.

Flow that was required:

1. Run the local labeling script → named all new communities, 0 placeholders left.
2. graphify cluster-only . --no-viz → re-clustered (community count shifted slightly), printed: "community set changed since labeling (N saved labels, M communities now; renamed X community(ies) by their hub). Run graphify label to refresh names with the LLM."
3. Had to re-run the local labeling script a second time to confirm 0 placeholders remained after the renumbering.

The tool's own suggested remedy (graphify label) is the LLM-cost path the project is explicitly trying to avoid — the message doesn't acknowledge that a non-LLM local labeling path exists or was just used.

Suggestion: make local/no-LLM community labeling (highest-degree-member naming) a first-class built-in mode of cluster-only, rather than something every project has to reimplement as an external script that then has to be manually re-run after every re-cluster. Bonus: cluster-only's own message could say "N communities need labels — run your local labeler or graphify label" instead of only surfacing the paid option.

4. "Surprising Connections" doesn't distinguish real insight from generic-symbol-name collisions

The generated report's Surprising Connections section included several entries like:

- `_complete_row()` --indirect_call--> `session()`  [INFERRED]
  some_worker.py → some_unrelated_test_fixture.py
- `_submit()` --indirect_call--> `session()`  [INFERRED]
  another_worker.py → same_unrelated_test_fixture.py (via a different caller)

Several unrelated production worker files apparently "indirect_call" into a session() defined in an unrelated test fixture. This is almost certainly a same-named-symbol collision (a common name like session appearing in many files) rather than a real dependency, but it's presented with the same framing/confidence as genuinely interesting cross-community bridges.

Suggestion: penalize (or bucket separately) INFERRED edges whose only evidence is a generic/common symbol name with no import or call-site proximity to back it up, so "surprising connections" trends toward real surprises instead of name collisions.

5. No way to get a real input/output token split from subagent-based semantic extraction

The skill's cost-tracking step expects input_tokens/output_tokens from each extraction subagent, written into the chunk JSON "using the real token counts from the Agent tool result's usage field." In practice (Claude Code host), the Agent tool's usage field only exposed a single combined token total — no input/output breakdown. Had to guess (input_tokens: 0, all as output), which makes graphify-out/cost.json wrong by construction for every subagent-driven (non-Gemini-API) extraction run.

Suggestion: either document a concrete way to derive the split when running under a host that only exposes a combined token count, or make the skill / cost.json schema tolerant of a single combined figure (tokens_total) instead of silently normalizing a guess into two fields that look precise but aren't.

6. The incremental-update flow is ~15 sequential manual steps, easy to get wrong

Following the /graphify --update skill instructions end-to-end for this run required manually running roughly 15 separate Python/bash blocks (detect → populate detect.json → code-only check → AST extract → cache check → chunk → dispatch subagent(s) → write token usage into the chunk file by hand → merge chunks → save cache → merge cached+new → merge AST+semantic → build_merge into existing graph → cluster/analyze/export → health check → label → cluster-only → export html → graph diff → save manifest/cost/cleanup). Two of the mistakes above (#2 the ignore-pattern quoting, #5 the token guess) happened during this manual sequence — both are exactly the kind of slip a long manual multi-step process invites.

Suggestion: collapse detect→extract→merge→cluster→report into a single CLI entrypoint (graphify update --exec, say) that only pauses for the semantic-extraction dispatch step (where a host-specific subagent hook is unavoidable), rather than requiring every intermediate JSON file to be shepherded by hand.

Environment

• graphify (graphifyy) via /graphify skill in Claude Code
• Existing .graphifyignore, graph.json with 12k+ nodes pre-update
• No GEMINI_API_KEY/GOOGLE_API_KEY set — semantic extraction ran via host subagent dispatch, not the Gemini backend