[Bug] Discuss agent prompt/source count and microcompact gating improvements

[MarkfuGod]

Problem or Motivation

I noticed two small agent-core issues after the v0.1.10 data-layer expansion, but since the PR template marks src/agent/ as a protected area, I wanted to open an issue first before submitting a PR.

1. The agent system prompt still says it has 7 data sources, while the loader registry now exposes 18 concrete sources after v0.1.10. This can make the agent under-advertise its current data capabilities.
2. AgentLoop currently runs Layer-1 _microcompact() every iteration, even when the transcript is well below any token pressure threshold. That clears older tool results to [cleared] on short runs, which can remove useful metrics, file contents, or search results the model may still need to reference.

Proposed Solution

I prepared a small draft branch here for maintainer review before opening a PR:

https://github.com/MarkfuGod/Vibe-Trading/tree/fix/agent-prompt-and-microcompact-gating

The draft does two things:

• Derives {data_source_count} from backtest.loaders.registry.VALID_SOURCES - {"auto"} instead of hardcoding 7, with a safe fallback so prompt building cannot break startup if registry import fails.
• Adds MICROCOMPACT_THRESHOLD = int(TOKEN_THRESHOLD * 0.5) and only calls _microcompact() when the estimated transcript size exceeds that threshold. Layer escalation then becomes: microcompact at ~20k, context collapse at ~28k, auto-compact at ~40k by default.

Alternatives Considered

• Just update 7 to 18: simpler, but it will drift again when sources are added or removed.
• Increase KEEP_RECENT: helps but still clears history every iteration, including short low-pressure runs.
• Move the fix into a larger context-management refactor: likely too broad for this small behavior correction.

Validation Done Locally

• Verified the dynamic count matches the registry:
  • data_source_count = 18
  • len(VALID_SOURCES - {"auto"}) = 18
• Targeted tests:
  • pytest tests/test_loop_helpers.py tests/test_context_attribution_layers.py -q -> 38 passed
  • Additional agent/memory-related targeted set -> 120 passed
• Broader local suite had unrelated environment failures around FastAPI 204 response handling / SPA deep-link tests, so I did not treat those as caused by this draft.

Contribution

I'm willing to submit a PR if maintainers agree this is an acceptable direction for the protected src/agent/ area.

[warren618]

Thanks for opening an issue first instead of going straight at src/agent/ — that's exactly the right call for the protected area, and much appreciated. I checked both points against the code and they're both real:

1. Stale data-source count — context.py:22 hardcodes 7 data sources while {skill_count} and {tool_count} right next to it are filled dynamically (context.py:187-189). len(VALID_SOURCES - {"auto"}) is currently 18, so the agent is under-advertising its data layer. Your fix (derive it from the registry, with a safe fallback on import error) is the right shape — please mirror the existing dynamic-count pattern.

2. Unconditional _microcompact() — confirmed: loop.py:559 runs it every iteration, while Layer 2 (collapse) and Layer 3 (auto-compact) are token-gated. With KEEP_RECENT=3, short low-pressure runs lose older tool results to [cleared] for no reason. Gating it behind MICROCOMPACT_THRESHOLD = int(TOKEN_THRESHOLD * 0.5) is a sound fix; keep the layer ordering intact (microcompact 20k → collapse 28k → auto-compact 40k).

Green-light to open the PR into src/agent/ for these two. To keep the protected-area review tight, please:

• Keep the PR scoped to exactly these two changes — no other agent-loop / context behavior.
• Add regression tests: data-source count derivation + the import-failure fallback path; and _microcompact is a no-op below the threshold but still fires above it. (Your test_loop_helpers.py / test_context_attribution_layers.py runs are a good base.)
• Make the data-source count derivation import-safe so a registry import error can never break prompt assembly / startup.
• Full suite green (pytest --ignore=agent/tests/e2e_backtest -q).

Tag me on the PR and I'll review promptly. Thanks for the careful diagnosis! 🙏

[MarkfuGod]

Thanks for the green light, @warren618!

I have opened the PR here: #296

Please take a look whenever you have a moment!