Skip to content

Context budget re-serializes message blocks on every LLM call #4848

Description

@Devesh36

Problem

context_budget.py estimates token usage by running json.dumps on message content blocks (especially tool results) just to get a length.

This runs before every LLM call in the ReAct loop, so during an investigation the same blocks can be serialized again and again as the transcript grows.

Tool schemas are already cached once per run. Message blocks are not.

Impact

  • Extra CPU on a hot path (per-iteration, per-investigation)
  • Worse with large tool results (logs, queries, etc.)
  • More noticeable on long investigations with many turns

Expected

Message token estimation should not re-pay full serialization cost for unchanged content on every think.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions