feat(memory): Rearchitect episodic tier for native chat conversation storage

[novaspectrayan]

Summary

Rearchitect the EPISODIC memory tier to natively store chat conversations as log-structured, variable-length experience records with a reinterpreted 64B header and inline CBOR document bodies — replacing the current anti-pattern of routing raw chat turns through the full cognitive ingestion pipeline (SEMANTIC tier + embedding + HNSW + BM25 + SPLADE + Hebbian + entity extraction).

ADR: ADR-0006 (Episodic Conversation Architecture)

User Story

As a Spector agent, I want my chat conversations to be stored efficiently with sub-millisecond append latency and O(1) paginated retrieval, so that I can maintain rich conversation history without polluting my semantic knowledge base or paying unnecessary cognitive pipeline overhead.

Acceptance Criteria

• EPISODIC tier uses reinterpreted 64B header: session_id (8B), role, sequence_id, body_length, timestamp — no Bloom filter, no importance, no emotional encoding
• Chat turns stored as 64B header + inline CBOR body (variable-length, log-structured)
• EpisodicLogMemory replaces fixed-stride EpisodicRecordMemory with append-only variable-length store
• EpisodicSessionIndex provides O(1) pagination (by session, cursor-based, tail-N for LLM context)
• Session index rebuilt on startup via single sequential mmap scan (sub-second for 100K records)
• Lightweight ingestion path: bypasses embedding, HNSW, BM25, SPLADE, Hebbian, entity extraction
• Chat text NOT indexed in shared text.dat (no BM25/SPLADE pollution)
• SpectorChatMemoryRepository.findByConversationId() fixed: uses browse (not vector recall)
• SpectorMemoryChatAdapter routes chat through storeEpisodic() instead of store()
• Circadian reflection trigger still fires after N episodic writes
• ConversationReflector reads from new episodic store and marks consolidated turns
• Full Maven build passes: mvn clean verify -T1C

Scope

IN scope (MVP):

• EpisodicFieldAccessor (header reinterpretation)
• ConversationRole enum
• EpisodicLogMemory (log-structured mmap store)
• EpisodicSessionIndex (pagination index)
• PartitionBundle EPISODIC region → variable-length
• Synapse integration (MemoryService, ChatAdapter, ChatMemoryRepository)
• ConversationReflector update

OUT of scope (future):

• Session manifest persistence (optimization for fast session listing)
• CBOR body compression
• Cross-partition episodic queries
• Episodic region compaction/GC

Getting Started

1. Build: cd /home/bharat/git/spector && mvn clean verify -T1C
2. Key files (memory module):
   • memory/spector-memory/.../kernel/layout/SynapticHeaderConstants.java
   • memory/spector-memory/.../cortex/EpisodicRecordMemory.java (to be replaced)
   • memory/spector-memory/.../kernel/bundle/PartitionBundle.java
   • memory/spector-memory/.../DefaultSpectorMemory.java
3. Key files (synapse module):
   • synapse/spector-synapse/.../agent/chat/infrastructure/SpectorMemoryChatAdapter.java
   • synapse/spector-synapse/.../memory/MemoryService.java
   • synapse/spector-synapse/.../agent/memory/SpectorChatMemoryRepository.java

RICE-C Score

Factor	Score	Rationale
Reach	5/5	Every Spector agent with chat uses this
Impact	3x	Sub-ms writes vs 30-80ms, eliminates SEMANTIC pollution
Confidence	90%	Architecture validated through code audit with CEO
Effort	L	5 phases, ~15 files, 2-3 weeks
Cognitive Fit	5/5	Core to Spector's neuroscience-inspired memory model

[titanspectrayan]

ADR-0006: Episodic Conversation Architecture

Status: Approved (CEO + CTO)
Author: @jarvis (CTO), @titan (Architect), Bharat (CEO)

Context

Chat conversations are currently stored through the full cognitive ingestion pipeline into the SEMANTIC tier. This causes:

• Performance: 30-80ms per turn (embedding + scoring + indexing)
• Pollution: BM25/SPLADE indexes contaminated with 'hello', 'thanks', 'ok'
• HNSW bloat: Chat vectors polluting the semantic vector index
• Incorrect recall: SpectorChatMemoryRepository.findByConversationId() uses vector recall with literal string 'session:ID' — a Bloom filter hit-rate bug

Decision

Repurpose MemoryType.EPISODIC as a log-structured conversation store:

1. 64B header retained (full cache line) with reinterpreted fields: valence→role, importance→sequence_id, synaptic_tags→session_id (raw 8B, NOT Bloom), encoding_surprise→body_length
2. CBOR document body inline with header (variable-length stride = 64 + body_length)
3. No vectorization — ConversationReflector handles semantic extraction via circadian reflection
4. Separate from text.dat — avoids BM25/SPLADE pollution
5. EpisodicSessionIndex for O(1) pagination, rebuilt on startup via sequential mmap scan
6. EpisodicLogMemory replaces fixed-stride EpisodicRecordMemory
7. Repurpose RegionId.EPISODIC(1) — no new region ID (confirmed by CEO)

Consequences

• EpisodicRecordMemory deprecated (no production data to migrate)
• CognitiveMemoryRouter needs episodic-aware write path
• Chat turns no longer appear in SEMANTIC recall results
• Circadian reflection remains the sole path for chat→semantic knowledge extraction