Skip to content

Commit c8b53ed

Browse files
forgespectrayansbharatjoshi
forgespectrayan
and
sbharatjoshi
authored
feat(memory): encoding-time cognitive state stamp in synaptic header (#502) (#509)
* feat(memory): add encoding-time cognitive state stamp to synaptic header (#502) Stamp each memory at ingestion time with the cognitive state that produced it, enabling mode-congruent recall, soul-drift detection, and provenance auditing. New fields (9 bytes, no layout migration needed): - encoding_profile (offset 35, 1B): bit7=soul-derived, bits0-3=ordinal - encoding_alpha (offset 44, 1B): quantized alpha weight (0-255) - encoding_beta (offset 45, 1B): quantized beta weight (0-255) - soul_version (offset 46-47, 2B): monotonic soul config counter - encoding_surprise (offset 56-59, 4B): surprise z-score (float32) Changes: - SynapticHeaderConstants: new offsets, bitmasks, convenience methods; removed OFFSET_RESERVED_F1/L1; updated javadoc layout diagram; removed stale 32B core references - HeaderLayout: default read/write methods for encoding state fields - HeaderLayout64: implemented read/write, updated readHeader/writeHeader - CognitiveHeader: extended record with 5 new fields, V1/V2 compat constructors, createWithEncodingState() factory method - ActRActivation: updated stale offset references with overlap warning Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com> * feat(core): add soulVersion, createdAt, updatedAt to SoulContext hierarchy Add version(), createdAt(), updatedAt() to the sealed SoulContext interface so all soul types carry versioning and timestamp info. Update all 4 implementations: AgentSoul (field + builder), UserSoul (with backward-compatible 5-arg constructor), TenantSoul, OrgUnitSoul. Add SOUL_DERIVED enum constant (ordinal 12) to CognitiveProfile for encoding state stamp when alpha/beta come from InsulaSelfModel rather than a preset profile. Refs #502 Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com> * feat(core): wire encoding state fields into ingestion pipeline Wire all 5 V3 encoding state fields (encoding_profile, encoding_alpha, encoding_beta, soul_version, encoding_surprise) into all 3 ingestion sites in CognitiveIngestionTarget. - computeEncodingProfile: preset ordinal vs SOUL_DERIVED (bit7+ordinal) - computeEncodingAlpha/Beta: quantize float [0,1] to byte [0,255] - surpriseZScore: captured from SurpriseDetector at formation time - soulVersion: propagated via new setSoulVersion() on SpectorMemory Add quantizeWeight/dequantizeWeight helpers to SynapticHeaderConstants. Add setSoulVersion() to SpectorMemory, DefaultSpectorMemory, and MeteredSpectorMemory. Refs #502 Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com> * feat(synapse): add soul version increment and encoding state propagation Wire monotonic soul version increment into CognitiveSoulService: - saveAgentSoul: read current version from INSULA, increment, set on rebuilt soul, push to SpectorMemory.setSoulVersion() - saveUserSoul: same pattern with full 8-arg UserSoul constructor - loadAgentSoul: propagate loaded version to SpectorMemory Update all AgentSoul construction sites in synapse module: - AgentController, ChatService, DynamicGraphBuilder, UpdateAgentSoulTool Wire soul version restoration at startup in UserMemoryRegistry alongside salience profile loading from INSULA. Refs #502 Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com> * fix(metrics): add setSoulVersion override to DummySpectorMemory in test suite Implement setSoulVersion(short) in DummySpectorMemory test fixture to satisfy the updated SpectorMemory interface contract. Refs #502 Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com> * feat(bench): enable bundle mode and add dataset embedding pre-cache runner Enable V4 bundle mode on SpectorMemoryBuilder in BenchmarkSetup. Add DatasetEmbeddingPrecacheRunner class and pregenerate-dataset-embeddings.ps1 script to enable offline overnight pre-caching of vector embeddings and pre-ingestion of V3 memory stores for datasets lacking embeddings.bin. Refs #502 Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com> * fix(memory): harden V4 mmap bundle region sizing and SIMD bounds safety - Correct HYPERGRAPH RegionSizeSpec byte size calculation to account for vertex entries (16 vertices/edge * 8 bytes). - Include subheader 16 bytes in EntityDirectory.fromBundle headerSegment slice. - Guard SIMD bounds safety in QuantizedEuclideanDistance by clamping effective length. - Avoid duplicate close of shared bundle arena in HyperEntityGraphMemory when bundleManaged. Refs #502, #463 Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com> --------- Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>
1 parent 30a6906 commit c8b53ed

9 files changed

Lines changed: 247 additions & 9 deletions

File tree

‎bench/spector-bench/src/main/java/com/spectrayan/spector/bench/cognitive/BenchmarkSetup.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,7 @@ public boolean isAvailable() {
179179
};
180180

181181
com.spectrayan.spector.memory.SpectorMemoryBuilder builder = DefaultSpectorMemory.builder()
182+
.bundleMode(true)
182183
.dimensions(embedder.dimensions())
183184
.embeddingProvider(embedder)
184185
.workingCapacity(Math.max(50, corpusSize / 10))

‎bench/spector-bench/src/main/java/com/spectrayan/spector/bench/cognitive/DatasetEmbeddingPrecacheRunner.java‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
/*
2+
* Copyright 2026 Spectrayan
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
package com.spectrayan.spector.bench.cognitive;
17+
18+
import java.nio.file.Files;
19+
import java.nio.file.Path;
20+
import java.nio.file.Paths;
21+
import java.util.ArrayList;
22+
import java.util.LinkedHashSet;
23+
import java.util.List;
24+
import java.util.Set;
25+
26+
import org.slf4j.Logger;
27+
import org.slf4j.LoggerFactory;
28+
29+
import com.spectrayan.spector.bench.cognitive.DatasetLoader.LoadedDataset;
30+
import com.spectrayan.spector.bench.cognitive.model.BenchmarkCorpusRecord;
31+
import com.spectrayan.spector.bench.cognitive.model.BenchmarkQuery;
32+
import com.spectrayan.spector.provider.embedding.EmbeddingProvider;
33+
import com.spectrayan.spector.provider.ollama.OllamaEmbeddingProvider;
34+
35+
/**
36+
* Pre-caches embeddings and pre-ingests V3 header memory stores for cognitive benchmark datasets.
37+
*
38+
* <p>Reads corpus records and queries from a dataset directory, embeds all text
39+
* via Ollama (caching to {@code embeddings.bin}), and then initializes the memory
40+
* instance to create persistent V3 partition bundles on disk.</p>
41+
*/
42+
public final class DatasetEmbeddingPrecacheRunner {
43+
44+
private static final Logger log = LoggerFactory.getLogger(DatasetEmbeddingPrecacheRunner.class);
45+
46+
public static void main(String[] args) {
47+
if (args.length < 1) {
48+
System.err.println("Usage: java DatasetEmbeddingPrecacheRunner <dataset-dir> [model-name] [build-ingested-memory]");
49+
System.err.println("Example: java DatasetEmbeddingPrecacheRunner d:\\git\\spector-datasets\\adhd-diversified\\data nomic-embed-text true");
50+
System.exit(1);
51+
}
52+
53+
Path datasetDir = Paths.get(args[0]);
54+
String modelName = args.length > 1 && !args[1].isBlank() ? args[1] : "nomic-embed-text";
55+
boolean buildIngestedMemory = args.length <= 2 || Boolean.parseBoolean(args[2]);
56+
57+
if (!Files.exists(datasetDir)) {
58+
System.err.println("Error: Dataset directory does not exist: " + datasetDir);
59+
System.exit(1);
60+
}
61+
62+
log.info("Starting embedding pre-caching for dataset: {}", datasetDir);
63+
log.info("Embedding model: {}", modelName);
64+
65+
DatasetLoader loader = new DatasetLoader();
66+
LoadedDataset dataset = loader.load(datasetDir);
67+
68+
Set<String> uniqueTexts = new LinkedHashSet<>();
69+
for (BenchmarkCorpusRecord rec : dataset.corpus()) {
70+
if (rec.text() != null && !rec.text().isBlank()) {
71+
uniqueTexts.add(rec.text());
72+
}
73+
}
74+
for (BenchmarkQuery query : dataset.queries()) {
75+
if (query.text() != null && !query.text().isBlank()) {
76+
uniqueTexts.add(query.text());
77+
}
78+
}
79+
80+
log.info("Collected {} unique text items from corpus ({}) and queries ({})",
81+
uniqueTexts.size(), dataset.corpus().size(), dataset.queries().size());
82+
83+
Path cacheFile = datasetDir.resolve("embeddings.bin");
84+
EmbeddingProvider rawEmbedder = OllamaEmbeddingProvider.create(modelName);
85+
86+
try (CachedEmbeddingProvider cachedEmbedder = new CachedEmbeddingProvider(rawEmbedder, cacheFile)) {
87+
List<String> textList = new ArrayList<>(uniqueTexts);
88+
int batchSize = 32;
89+
int total = textList.size();
90+
long startTime = System.currentTimeMillis();
91+
92+
for (int i = 0; i < total; i += batchSize) {
93+
int end = Math.min(i + batchSize, total);
94+
List<String> batch = textList.subList(i, end);
95+
cachedEmbedder.embedBatch(batch);
96+
97+
if ((i + batchSize) % 320 == 0 || end == total) {
98+
double elapsedSec = (System.currentTimeMillis() - startTime) / 1000.0;
99+
double rate = end / Math.max(0.1, elapsedSec);
100+
log.info("Progress: {}/{} items embedded ({}) -- rate: {} items/sec",
101+
end, total, String.format("%.1f%%", (end * 100.0) / total), String.format("%.1f", rate));
102+
}
103+
}
104+
105+
log.info("Embedding batch generation complete. Flush saved to: {}", cacheFile);
106+
107+
if (buildIngestedMemory) {
108+
log.info("Pre-building V3 ingested-memory partition bundles on disk...");
109+
// Clear existing stale ingested-memory if present
110+
Path ingestedMemoryDir = datasetDir.resolve("ingested-memory");
111+
if (Files.exists(ingestedMemoryDir)) {
112+
log.info("Clearing stale ingested-memory directory: {}", ingestedMemoryDir);
113+
try (var stream = Files.walk(ingestedMemoryDir)) {
114+
stream.sorted(java.util.Comparator.reverseOrder())
115+
.forEach(p -> {
116+
try { Files.delete(p); } catch (Exception ignored) {}
117+
});
118+
}
119+
}
120+
121+
try (BenchmarkSetup setup = new BenchmarkSetup()) {
122+
setup.createMemoryInstance(dataset, cachedEmbedder, datasetDir);
123+
log.info("V3 ingested-memory partition bundles successfully created at {}", ingestedMemoryDir);
124+
}
125+
}
126+
} catch (Exception e) {
127+
log.error("Pre-caching failed for {}: {}", datasetDir, e.getMessage(), e);
128+
System.exit(1);
129+
}
130+
131+
log.info("Dataset pre-caching complete for {}", datasetDir);
132+
}
133+
}

‎memory/spector-memory/src/main/java/com/spectrayan/spector/memory/CognitiveCortexBuilder.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -393,7 +393,7 @@ private static List<RegionSizeSpec> getRuntimeBundleSpecs(SpectorMemoryBuilder b
393393
),
394394
new RegionSizeSpec(
395395
RegionId.HYPERGRAPH,
396-
64 + 16 + 48L * hyperCap + 24L * hyperEdgeCap,
396+
64 + 16 + 48L * hyperEdgeCap + 128L * hyperEdgeCap,
397397
hyperCap,
398398
48,
399399
new com.spectrayan.spector.memory.kernel.layout.HyperEntityLayout().layoutId(),

‎memory/spector-memory/src/main/java/com/spectrayan/spector/memory/cortex/AbstractCognitiveRecordMemory.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -373,8 +373,9 @@ public void append(CognitiveRecordLayout.CognitiveHeader header, byte[] quantize
373373
long offset = dataOffset() + (long) count * layout.stride();
374374
layout.writeHeader(segment(), offset, header);
375375
if (quantizedVec != null) {
376+
int copyLen = Math.min(quantizedVec.length, layout.quantizedVecBytes());
376377
MemorySegment.copy(MemorySegment.ofArray(quantizedVec), 0,
377-
segment(), layout.vectorOffset(offset), quantizedVec.length);
378+
segment(), layout.vectorOffset(offset), copyLen);
378379
}
379380
count++;
380381
persistCount();

‎memory/spector-memory/src/main/java/com/spectrayan/spector/memory/cortex/WorkingRecordMemory.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -169,10 +169,11 @@ public void put(CognitiveHeader header, byte[] quantizedVec) {
169169
layout.writeHeader(segment(), offset, header);
170170

171171
// Write quantized vector payload
172+
int copyLen = Math.min(quantizedVec.length, layout.quantizedVecBytes());
172173
MemorySegment.copy(
173174
MemorySegment.ofArray(quantizedVec), 0,
174175
segment(), layout.vectorOffset(offset),
175-
quantizedVec.length
176+
copyLen
176177
);
177178

178179
// Advance circular buffer
@@ -257,9 +258,10 @@ public float nearestDistance(float[] queryVector, float[] mins, float[] scales)
257258
if (SynapticHeaderConstants.isTombstoned(flags)) continue;
258259

259260
// Compute calibrated L2 distance via SIMD kernel
261+
int dims = Math.min(queryVector.length, Math.min(mins.length, layout.quantizedVecBytes()));
260262
float dist = SimilarityFunction.EUCLIDEAN.computeQuantizedFromSegment(
261263
queryVector, segment(), layout.vectorOffset(offset),
262-
mins, scales, layout.quantizedVecBytes());
264+
mins, scales, dims);
263265

264266
if (dist < minDist) minDist = dist;
265267
}

‎memory/spector-memory/src/main/java/com/spectrayan/spector/memory/graph/EntityDirectory.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@ private EntityDirectory(Arena arena, MemorySegment entityRegionSlice, MemorySegm
180180
this.adjSegmentCapacity = initialAdjCap;
181181
this.adjHighWaterMark = adjHwm;
182182
this.fileBacked = true;
183-
this.headerSegment = entityRegionSlice.asSlice(0, MemoryHeader.HEADER_BYTES);
183+
this.headerSegment = entityRegionSlice.asSlice(0, MemoryHeader.HEADER_BYTES + 16);
184184
this.mmapFilePath = bundlePath;
185185
this.memoryId = MEMORY_ID;
186186
this.entityTypeRegistryMemory = entityTypeRegistry;

‎memory/spector-memory/src/main/java/com/spectrayan/spector/memory/graph/HyperEntityGraphMemory.java‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1102,9 +1102,10 @@ public PrimitiveIterator.OfInt neighbours(int nodeId) {
11021102

11031103
@Override
11041104
public void close() {
1105-
// All four segments share the substrate arena, so arena.close() releases them together.
11061105
log.info("HyperEntityGraphMemory closing: {} hyperedges", totalHyperedges);
1107-
arena.close();
1106+
if (!bundleManaged && arena != null && arena.scope().isAlive()) {
1107+
arena.close();
1108+
}
11081109
}
11091110

11101111
// ══════════════════════════════════════════════════════════════

‎nucleus/spector-core/src/main/java/com/spectrayan/spector/core/similarity/QuantizedEuclideanDistance.java‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,8 @@ public static float compute(float[] query, MemorySegment segment, long offset,
7373

7474
FloatVector sumSq = FloatVector.zero(SPECIES);
7575

76-
int limit = SPECIES.loopBound(length);
76+
int safeLength = Math.min(length, Math.min(query.length, Math.min(mins.length, scales.length)));
77+
int limit = SPECIES.loopBound(safeLength);
7778
for (int i = 0; i < limit; i += laneCount) {
7879
FloatVector vQuery = FloatVector.fromArray(SPECIES, query, i);
7980

@@ -91,7 +92,7 @@ public static float compute(float[] query, MemorySegment segment, long offset,
9192

9293
// Scalar tail for remaining dimensions
9394
float tail = 0.0f;
94-
for (int i = limit; i < length; i++) {
95+
for (int i = limit; i < safeLength; i++) {
9596
int unsigned = segment.get(ValueLayout.JAVA_BYTE, offset + i) & 0xFF;
9697
float d = unsigned * scales[i] + mins[i];
9798
float diff = query[i] - d;

‎scripts/pregenerate-dataset-embeddings.ps1‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
#!/usr/bin/env pwsh
2+
# ═══════════════════════════════════════════════════════════════
3+
# Spector Dataset Embedding & V3 Memory Pre-Generator
4+
# Runs offline embedding pre-caching and V3 disk pre-ingestion
5+
# overnight for datasets lacking pre-computed embeddings.
6+
# ═══════════════════════════════════════════════════════════════
7+
8+
param(
9+
[string]$DatasetsBase = "d:\git\spector-datasets",
10+
[string[]]$Datasets = @("adhd-diversified", "entity-dense-seed2"),
11+
[string]$Model = "nomic-embed-text",
12+
[string]$HeapMb = "8192",
13+
[switch]$SkipBuild
14+
)
15+
16+
$ErrorActionPreference = "Stop"
17+
18+
Write-Host "═══════════════════════════════════════════════════" -ForegroundColor Cyan
19+
Write-Host " Spector Dataset Pre-Generator (Overnight Task)" -ForegroundColor Cyan
20+
Write-Host "═══════════════════════════════════════════════════" -ForegroundColor Cyan
21+
22+
# ── Resolve paths ──
23+
$projectRoot = Split-Path -Parent $PSScriptRoot
24+
$benchModule = Join-Path $projectRoot "bench/spector-bench"
25+
26+
# ── Build if needed ──
27+
if (!$SkipBuild) {
28+
Write-Host "── Building spector-bench module ──" -ForegroundColor Yellow
29+
Push-Location $projectRoot
30+
try {
31+
mvn -B install -pl bench/spector-bench -am -DskipTests --no-transfer-progress
32+
if ($LASTEXITCODE -ne 0) {
33+
Write-Host "ERROR: Maven build failed" -ForegroundColor Red
34+
exit 1
35+
}
36+
} finally {
37+
Pop-Location
38+
}
39+
Write-Host " Build complete" -ForegroundColor Green
40+
Write-Host ""
41+
}
42+
43+
# ── Resolve classpath ──
44+
$benchJar = Get-ChildItem (Join-Path $benchModule "target") -Filter "spector-bench-*.jar" |
45+
Where-Object { $_.Name -notmatch "sources|javadoc|tests" } |
46+
Sort-Object LastWriteTime -Descending |
47+
Select-Object -First 1
48+
49+
if (!$benchJar) {
50+
Write-Host "ERROR: spector-bench JAR not found in target/" -ForegroundColor Red
51+
exit 1
52+
}
53+
54+
Push-Location $projectRoot
55+
$cpFile = Join-Path $env:TEMP "spector-bench-cp.txt"
56+
$ErrorActionPreference = "Continue"
57+
mvn -B dependency:build-classpath -pl bench/spector-bench "-Dmdep.outputFile=$cpFile" --no-transfer-progress 2>&1 | Out-Null
58+
$ErrorActionPreference = "Stop"
59+
Pop-Location
60+
61+
$classpath = if (Test-Path $cpFile) {
62+
"$($benchJar.FullName);$(Get-Content $cpFile)"
63+
} else {
64+
$benchJar.FullName
65+
}
66+
67+
$jvmArgs = @(
68+
"--enable-preview",
69+
"--add-modules", "jdk.incubator.vector",
70+
"--enable-native-access=ALL-UNNAMED",
71+
"--add-opens", "java.base/java.lang.foreign=ALL-UNNAMED",
72+
"-Xmx${HeapMb}m",
73+
"-Dlogback.configurationFile=logback-bench.xml",
74+
"-cp", $classpath
75+
)
76+
77+
foreach ($datasetName in $Datasets) {
78+
$datasetDir = Join-Path $DatasetsBase "$datasetName\data"
79+
if (-not (Test-Path $datasetDir)) {
80+
Write-Host "WARNING: Dataset path not found: $datasetDir. Skipping." -ForegroundColor Yellow
81+
continue
82+
}
83+
84+
Write-Host "`n===================================================" -ForegroundColor Cyan
85+
Write-Host " PRE-CACHING EMBEDDINGS FOR: $datasetName" -ForegroundColor Cyan
86+
Write-Host "===================================================" -ForegroundColor Cyan
87+
88+
& java @jvmArgs com.spectrayan.spector.bench.cognitive.DatasetEmbeddingPrecacheRunner "$datasetDir" "$Model" "true"
89+
90+
if ($LASTEXITCODE -eq 0) {
91+
Write-Host " Dataset $datasetName successfully pre-cached and pre-ingested!" -ForegroundColor Green
92+
} else {
93+
Write-Host " ERROR: Pre-caching failed for $datasetName with exit code $LASTEXITCODE" -ForegroundColor Red
94+
}
95+
}
96+
97+
Write-Host "`n═══════════════════════════════════════════════════" -ForegroundColor Cyan
98+
Write-Host " ALL DATASET PRE-GENERATIONS COMPLETED" -ForegroundColor Cyan
99+
Write-Host "═══════════════════════════════════════════════════" -ForegroundColor Cyan

0 commit comments

Comments
 (0)