[SMK Phase 3] Codec Pipeline Formalization, Inline Migrator Purge & Per-Record CRC32C

[novaspectrayan]

Problem Statement

Phase 3 of the Spector Memory Kernel (SMK) roadmap formalizes the Codec migration pipeline per spector-memory-codec-design.md, eliminates 5 ad-hoc inline migrators scattered across memory subsystems, and implements mandatory per-record CRC32C validation (Kernel Invariant #3).

Target Architecture & Codec Chain Flow

sequenceDiagram
    participant F as SpectorMemoryFactory
    participant C as Codecs.ensureCurrent
    participant D as FormatDetector
    participant CH as CodecChain
    participant S as CodecStep(s)
    participant M as AbstractMemory (mmap)

    F->>C: ensureCurrent(registry, id, layout, path, enc, sidecars)
    C->>D: detect(path)
    D-->>C: FormatId (e.g. TPCH v1 / SMKM v2)
    alt needs migration
        C->>CH: run(ctx)
        loop until current
            CH->>S: apply(ctx) [RewriteFileStep / InPlaceHeaderStep]
        end
        CH-->>C: MigrationResult
    end
    F->>M: new XxxMemory(path, ...)

Loading

Scope & Technical Tasks

1. Kernel Codec Engine (kernel/codec/):

   • FormatId: Models magic + version (FormatId.smkm(version) or legacy magic).
   • FormatDetector: Detects file magic & schema version.
   • CodecStep: Sealed interface (InPlaceHeaderStep, RewriteFileStep, IdentityStep).
   • MigrationContext & MigrationResult: Encapsulates path, layout, encryptor, sidecars, and backup retention.
   • CodecChain: Eager validation of migration steps, cycle detection, and DAG execution.
   • CodecRegistry: Registration map for codecs by layoutId and legacy magic.
   • Codecs.ensureCurrent: Single static entry point before mapping.

2. Per-Record CRC32C Validation (Invariant build(deps): bump actions/setup-java from 4 to 5 #3):

   • Implement record payload CRC32C checksum write & verification in AbstractRecordMemory.
   • Enable crcEnabled() = true in layout implementations.

3. Migrator Step Extractions:

   • TpchToSmkmStep (TemporalChainMemory)
   • LegacyTextToSmkmStep (TextAppendMemory)
   • TregToSmkmStep (TypeRegistryMemory)
   • MidxToSmkmStep (IndexRecordMemory + .idpl sidecar)
   • HgphToCsrStep (HebbianGraphMemory)
   • Purge inline migrator methods from all 5 domain memory classes.

Acceptance Criteria

• kernel/codec/ engine passes eager chain validation and crash-safety tests.
• All 5 inline legacy migrators are replaced by formal CodecStep classes.
• AbstractRecordMemory verifies per-record CRC32C checksums.
• CodecChainTest and RecordCrcVerificationTest pass with 100% success rate.
• mvn clean install -DskipTests passes across all 26 reactor modules.
• mvn test -pl nucleus/spector-test-support,memory/spector-memory passes with 0 failures and 0 errors.

Priority & RICE-C Score

• Reach: 10/10 (Core Kernel Persistence)
• Impact: 10/10 (Crash Safety & Format Integrity)
• Confidence: 100%
• Effort: 2 Person-Days
• Score: CRITICAL

[titanspectrayan]

🏛️ Titan Architectural Review & Codec Charter Alignment (Issue #409)

Status: APPROVED (Target Codec Engine Architecture Verified)

1. Codec Engine Sequence Flow

sequenceDiagram
    participant F as SpectorMemoryFactory
    participant C as Codecs.ensureCurrent
    participant D as FormatDetector
    participant CH as CodecChain
    participant S as CodecStep(s)
    participant M as AbstractMemory (mmap)

    F->>C: ensureCurrent(registry, id, layout, path, enc, sidecars)
    C->>D: detect(path)
    D-->>C: FormatId (e.g. TPCH v1 / SMKM v2)
    alt needs migration
        C->>CH: run(ctx)
        loop until current
            CH->>S: apply(ctx) [RewriteFileStep / InPlaceHeaderStep]
        end
        CH-->>C: MigrationResult
    end
    F->>M: new XxxMemory(path, ...)

Loading

2. Key Architectural Invariants

• FormatId & FormatDetector: Unifies foreign magics (TPCH, MIDX, HGPH, EGMM, TREG) and SMKM schema versions into a single migration hop abstraction (FormatId → FormatId).
• Crash-Safe Rewrite Protocol: RewriteFileStep enforces atomic temp file write → fsync → .bak.vN creation → ATOMIC_MOVE → parent dir fsync. Prevents file truncation and corruption windows on power loss.
• Per-Record CRC32C (Invariant build(deps): bump actions/setup-java from 4 to 5 #3): Implemented in AbstractRecordMemory to validate record payload integrity on write and read.