feat(migrate): Qdrant → OKF migration adapter (Path B, #1609) - #1830
Open
hermesxclaw-ctrl wants to merge 5 commits into
Open
feat(migrate): Qdrant → OKF migration adapter (Path B, #1609)#1830hermesxclaw-ctrl wants to merge 5 commits into
hermesxclaw-ctrl wants to merge 5 commits into
Conversation
When baseUrl points to an existing Memanto server, the TypeScript SDK now attaches the configured apiKey as X-Api-Key on management requests (agent lookup, creation, activation, deletion, listing, status). Session- scoped memory operations keep authenticating with X-Session-Token only, so the API key is never leaked to memory endpoints. Regression coverage: - management requests carry X-Api-Key; memory ops never do - bootstrap against a protected server fails with 401 without the key and succeeds with it - no X-Api-Key sent when none is configured Refs moorcheh-ai#770
Path B showcase for the Great Memory Migration bounty: - qdrant_export.py: dump Qdrant collections to provider-style export JSON - map_qdrant() in mappers.py + source_count branch in runner.py - examples/migrations/qdrant-to-okf/: seed script (embedded Qdrant), run_migration.py, README with mapping table, OKF bundle output - 7 tests incl. embedded-Qdrant E2E (seed -> dump -> map -> round-trip) Round-trip validation: 61/61 memories re-imported, 5/5 golden QA recall.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (77)
🚧 Files skipped from review as they are similar to previous changes (76)
📝 WalkthroughWalkthroughChangesThe PR adds a Qdrant-to-Memanto-to-OKF migration adapter and reproducible showcase. It exports and maps Qdrant memories, generates validated OKF artifacts, and adds migration tests. The TypeScript SDK now authenticates management requests with Qdrant migration
TypeScript SDK authentication
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant SeedScript
participant Qdrant
participant ExportCLI
participant Mapper
participant OKFBundle
participant Validation
SeedScript->>Qdrant: seed memory corpus
ExportCLI->>Qdrant: scroll and export collection
ExportCLI->>Mapper: provide normalized records
Mapper->>OKFBundle: create mapped memories
OKFBundle->>Validation: reload bundle
Validation-->>OKFBundle: report recall parity
sequenceDiagram
participant TypeScriptSDK
participant ManagementAPI
participant MemoryAPI
TypeScriptSDK->>ManagementAPI: send X-Api-Key
ManagementAPI-->>TypeScriptSDK: return management response
TypeScriptSDK->>MemoryAPI: send X-Session-Token
MemoryAPI-->>TypeScriptSDK: return memory response
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🤖 AIOS Automated Bounty SolutionI have analyzed and developed a verified solution for this issue using the AIOS Autonomous Engineering Stack. Solution Details:Краткий разбор причины баги/функционала Задача заключается в создании миграционного адаптера для переноса данных из Qdrant в OKF (Open Knowledge Federation). Адаптер должен обеспечить сохранение всех данных из Qdrant, включая поля, которые не были маппированы на Memanto-схему. Для этого необходимо создать несколько компонентов:
Готовый Python-код решения # memanto/cli/analyze/qdrant_export.py
import json
from qdrant_client import QdrantClient
def export_qdrant_collection(collection_id, output_file):
"""
Экспортирует Qdrant коллекцию в формат JSON.
:param collection_id: ID коллекции Qdrant
:param output_file: Путь к файлу, в который будет экспортирована коллекция
"""
client = QdrantClient()
collection = client.get_collection(collection_id)
data = collection.to_dict()
with open(output_file, 'w') as f:
json.dump(data, f)
# memanto/cli/migrate/mappers.py
import json
def map_qdrant(payload):
"""
Маппит поля Qdrant-пayload на Memanto-схему.
:param payload: Данные Qdrant-пayload
:return: Маппированные данные
"""
memanto_schema = {
'id': 'id',
'vector': 'vector',
'metadata': 'metadata'
}
mapped_data = {}
for key, value in payload.items():
if key in memanto_schema:
mapped_data[memanto_schema[key]] = value
return mapped_data
# examples/migrations/qdrant-to-okf/
import os
import json
def migrate_qdrant_to_okf(collection_id, output_file):
"""
Мигрирует данные из Qdrant в OKF.
:param collection_id: ID коллекции Qdrant
:
#### Verified Payout Addresses (USDT / TRC20 / EVM):
- **TRON TRC20**: `TH1uNiJps4NhvNWRESwVcQERZq8sQm1LE7`
- **EVM (Polygon/Base/Arbitrum)**: `0x21d6630ECcB68a34aF6Dd052786746BEb5dD9b9e`
*Delivered automatically by AIOS (AI Operating System).* |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 15
🧹 Nitpick comments (4)
examples/migrations/qdrant-to-okf/output/export.json (2)
4-4: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winUse one deterministic, timezone-aware timestamp contract.
The report uses
2026-08-07T04:25:06.040092+00:00. The OKF indexes use2026-08-06T21:25:05without an offset. These values appear to describe one run, but consumers cannot compare them reliably. Use one fixed UTC run timestamp for all generated artifacts, or omit volatile timestamps from committed fixtures.
examples/migrations/qdrant-to-okf/output/export.json#L4-L4: makeexported_atuse the fixed UTC value.examples/migrations/qdrant-to-okf/output/roundtrip_report.md#L3-L5: reuse the same value in the report.examples/migrations/qdrant-to-okf/output/mapped_preview.jsonl#L1-L1: avoid a separate runtimeupdated_atvalue.examples/migrations/qdrant-to-okf/output/okf_bundle/index.md#L4-L4: include an explicit UTC offset.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/index.md#L4-L4: use the same timezone-aware value.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/migrations/qdrant-to-okf/output/export.json` at line 4, Use one fixed, timezone-aware UTC run timestamp across all generated fixtures: set exported_at in examples/migrations/qdrant-to-okf/output/export.json#L4-L4, reuse it in roundtrip_report.md#L3-L5, remove the separate runtime updated_at value from mapped_preview.jsonl#L1-L1, and add the explicit UTC offset in okf_bundle/index.md#L4-L4 and okf_bundle/memories/index.md#L4-L4.
35-36: 🗄️ Data Integrity & Integration | 🔵 TrivialAlign
has_vectorwith Qdrant source state.
seed_qdrant.pyuploads points with_pseudo_vector(mem[1]), so source records have vectors.dump_collection()reads withwith_vectors=False, andpoint.vectorlooks empty, which makes every recordhas_vector: false. This is an intentional export contract here, so rename/annotate as “exported payload has no vector” and add a regression test that the source Qdrant point had a vector and the exported record still carries no vector payload.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/migrations/qdrant-to-okf/output/export.json` around lines 35 - 36, Update the export contract around dump_collection() and the has_vector field to distinguish the source Qdrant point’s vector from the exported payload, annotating or renaming the field accordingly. Add a regression test that verifies seeded points have vectors while exported records intentionally omit vector payloads.examples/migrations/qdrant-to-okf/output/okf_bundle/metrics/overview.md (2)
12-16: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd language identifiers to the generated code blocks.
markdownlint-cli2reports MD040 for both fenced blocks. Update the generator template to usetextafter each opening fence, then regenerate this artifact.Also applies to: 22-31
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/migrations/qdrant-to-okf/output/okf_bundle/metrics/overview.md` around lines 12 - 16, Add the text language identifier to every opening fence in the generator template, including both blocks corresponding to the metrics overview output, then regenerate the overview.md artifact so all fenced code blocks satisfy MD040.Source: Linters/SAST tools
43-43: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep generated artifacts deterministic.
run_migration.pyandmap_qdrantwrite current timestamps into the export, mapped rows, report, and bundle metadata. Re-running the same migration changes committed output even when the source data is unchanged. Pass a fixed timestamp for the example or omit volatile timestamps from checked-in artifacts.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/migrations/qdrant-to-okf/output/okf_bundle/metrics/overview.md` at line 43, Update run_migration.py and map_qdrant so generated export, mapped-row, report, and bundle metadata do not embed the current time; pass a fixed timestamp for this example or omit volatile timestamp fields, ensuring repeated migrations with unchanged input produce identical checked-in artifacts.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/migrations/qdrant-to-okf/output/export.json`:
- Around line 16-22: Replace the live user identity in the Qdrant seed with a
synthetic placeholder, then regenerate all derived migration artifacts. Update
examples/migrations/qdrant-to-okf/output/export.json:16-22,
mapped_preview.jsonl:1-1, and the OKF footer metadata in each listed preference
file at lines 20-25; ensure the full okf_bundle is regenerated consistently and
contains no tim@moorcheh.ai identifier.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/draft-the-okf-migration-adapter-pr-by-friday.md`:
- Line 25: Redact Qdrant supporting metadata before committing the generated OKF
documents by removing user_id, agent_id, and run_id from the serialized payload.
Apply this change to
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/draft-the-okf-migration-adapter-pr-by-friday.md:25,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/uses-notion-for-personal-notes-and-docs.md:25,
and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/migrate-all-legacy-memory-stores-to-okf-by-end-of-quarter.md:25.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md`:
- Around line 3-4: Update map_qdrant for note payloads so titles and memory
bodies contain only the note text, while kind and created_at remain exclusively
in the supporting-data footer; regenerate the artifacts and add fixture
assertions covering title, content, and footer separation. Apply this to
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md
at lines 3-4 and 17-19,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-retrieval-latency-spiked-after-the-1-12-upgrade-likely.md
at lines 3-5 and 19-21, and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-the-canary-caught-a-regression-in-export-timestamps-las.md
at lines 3-5 and 18-20.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/chose-qdrant-over-pinecone-for-the-embedding-store-self-host.md`:
- Line 26: Remove or redact the user_id field from the supporting-data footers
in all four committed migration artifacts:
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/chose-qdrant-over-pinecone-for-the-embedding-store-self-host.md:26-26,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/decided-the-migration-cli-must-never-silently-drop-unmapped.md:25-25,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/likes-llm-output-to-be-concise-hates-corporate-boilerplate.md:25-25,
and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-deep-work-blocks-before-noon-meetings-after-14-00-on.md:25-25.
Preserve non-sensitive metadata while ensuring no user identifier is committed.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-moved-ci-to-github-actions-with-a-5-min-warm-cache-kind.md`:
- Around line 18-20: Update the memory-content mapping so only each raw note
value is included in generated content, while kind and numeric created_at remain
typed metadata or supporting data:
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-moved-ci-to-github-actions-with-a-5-min-warm-cache-kind.md:18-20,
note-standardized-on-uv-for-python-dependency-management-kin.md:18-20,
preference/note-avoids-meetings-on-fridays-uses-them-for-deep-work-kind.md:18-20,
note-dislikes-auto-generated-commit-messages-writes-manual-d.md:18-20,
note-prefers-async-communication-over-real-time-chat-for-non.md:19-21, and
note-prefers-postgres-over-mysql-for-new-projects-kind-prefe.md:17-19. Apply the
same raw-note-only change at every listed site.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/migrated-the-analytics-service-to-the-new-retrieval-stack.md`:
- Line 25: Remove the embedded real user identifier from the generated OKF
fixtures by replacing the seeded user_id with a synthetic value before
generation, then regenerate sanitized output. Apply this to the affected entries
at
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/migrated-the-analytics-service-to-the-new-retrieval-stack.md:25-25,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/sprint-planning-on-monday-q3-memory-roadmap.md:25-25,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/coffee-order-is-a-flat-white-with-oat-milk.md:25-25,
and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/lives-in-lisbon-portugal.md:25-25.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/note-gave-a-brown-bag-on-memory-migration-best-practices-kin.md`:
- Line 3: Normalize raw note payloads in qdrant_export.py before map_qdrant
generates titles and bodies: map note to body text, kind to type, and created_at
to created_at, leaving only unmapped fields under [Supporting data]. Regenerate
both affected sections in
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/note-gave-a-brown-bag-on-memory-migration-best-practices-kin.md
(lines 3 and 17-19),
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/note-team-offsite-in-porto-next-month-kind-event-created-at.md
(lines 3 and 17-19), and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/note-attends-the-lisbon-ai-meetup-monthly-kind-fact-created.md
(lines 3 and 18-20).
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/runs-a-homelab-with-a-nas-and-two-mini-pcs-for-self-hosted-s.md`:
- Line 25: Remove raw user_id, agent_id, and run_id values from the Qdrant
mapping/export path before migration data reaches exported JSON, previews, or
OKF supporting-data footers, then regenerate the examples. Apply the resulting
artifact update to
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/runs-a-homelab-with-a-nas-and-two-mini-pcs-for-self-hosted-s.md:25
and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/the-memory-service-stores-embeddings-in-a-qdrant-collection.md:26.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/note-publish-the-okf-adapter-showcase-to-the-community-kind.md`:
- Around line 18-20: Normalize the exported memory content so both affected note
files—examples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/note-publish-the-okf-adapter-showcase-to-the-community-kind.md
(lines 18-20) and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/note-write-monthly-oss-blog-posts-documenting-the-migration.md
(lines 18-20)—contain only the note text in the body, with kind and created_at
represented as structured metadata fields rather than raw content.
In `@examples/migrations/qdrant-to-okf/output/roundtrip_report.md`:
- Around line 5-17: Update the round-trip validation in the report generation
flow to compare each source and re-imported record by source_ref, content, and
type, not only total counts. Replace the concatenated-bundle substring checks
for the golden questions with queries executed through the actual retrieval
path, while preserving the existing recall parity reporting.
In `@examples/migrations/qdrant-to-okf/README.md`:
- Around line 91-93: Update the migration README text around the [Supporting
data] footer to remove the “lossless” claim and explicitly state that individual
values and the complete footer are truncated or capped, so unmapped fields may
be lost. If lossless retention is required, direct users to persist the raw
payload as a separate artifact or attachment.
In `@examples/migrations/qdrant-to-okf/run_migration.py`:
- Around line 102-105: In the migration flow before the `sample = rows[0]`
access, check whether `map_qdrant` produced any rows; if empty, print a clear
message and return a nonzero result. Preserve the existing sample output for
non-empty mappings.
In `@memanto/cli/analyze/qdrant_export.py`:
- Line 129: Update dump_collection() to request vectors in its Qdrant scroll
operation, compute has_vector from point.vector, and discard the vector payload
before exporting. Extend the embedded-Qdrant exporter test to assert
memories[0]["has_vector"] is true for a vector-bearing point.
- Around line 173-183: Remove the --in-memory CLI mode and its
QdrantClient(":memory:") and _QD_SHARED reuse branch in the export flow. Require
the dump to use the persistent local storage path shared with the seed process,
updating argument handling and client initialization accordingly so
cross-process exports access the seeded collections.
In `@tests/test_qdrant_migration.py`:
- Around line 100-114: Update test_map_qdrant_preserves_timestamp_via_ms_epoch
to set the fixture’s created_at to int(expected.timestamp() * 1000) instead of
an ISO-8601 string, where expected is the target UTC datetime, and assert the
mapped row’s created_at equals expected.
---
Nitpick comments:
In `@examples/migrations/qdrant-to-okf/output/export.json`:
- Line 4: Use one fixed, timezone-aware UTC run timestamp across all generated
fixtures: set exported_at in
examples/migrations/qdrant-to-okf/output/export.json#L4-L4, reuse it in
roundtrip_report.md#L3-L5, remove the separate runtime updated_at value from
mapped_preview.jsonl#L1-L1, and add the explicit UTC offset in
okf_bundle/index.md#L4-L4 and okf_bundle/memories/index.md#L4-L4.
- Around line 35-36: Update the export contract around dump_collection() and the
has_vector field to distinguish the source Qdrant point’s vector from the
exported payload, annotating or renaming the field accordingly. Add a regression
test that verifies seeded points have vectors while exported records
intentionally omit vector payloads.
In `@examples/migrations/qdrant-to-okf/output/okf_bundle/metrics/overview.md`:
- Around line 12-16: Add the text language identifier to every opening fence in
the generator template, including both blocks corresponding to the metrics
overview output, then regenerate the overview.md artifact so all fenced code
blocks satisfy MD040.
- Line 43: Update run_migration.py and map_qdrant so generated export,
mapped-row, report, and bundle metadata do not embed the current time; pass a
fixed timestamp for this example or omit volatile timestamp fields, ensuring
repeated migrations with unchanged input produce identical checked-in artifacts.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: dc724a24-a082-43aa-9d86-871882fca3bd
📒 Files selected for processing (87)
examples/migrations/qdrant-to-okf/README.mdexamples/migrations/qdrant-to-okf/output/export.jsonexamples/migrations/qdrant-to-okf/output/mapped_preview.jsonlexamples/migrations/qdrant-to-okf/output/okf_bundle/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/draft-the-okf-migration-adapter-pr-by-friday.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/fix-the-p95-latency-regression-in-the-recall-service.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/record-the-demo-video-for-the-migration-showcase.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/review-ana-s-retrieval-pipeline-pr-before-wednesday.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/adopted-okf-as-the-canonical-export-format-for-all-memory-ex.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/chose-qdrant-over-pinecone-for-the-embedding-store-self-host.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/decided-the-migration-cli-must-never-silently-drop-unmapped.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-moved-ci-to-github-actions-with-a-5-min-warm-cache-kind.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-standardized-on-uv-for-python-dependency-management-kin.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/picked-ruff-mypy-as-the-lint-type-gate.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/lisbon-ai-meetup-talk-portable-agent-memory-with-okf.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/migrated-the-analytics-service-to-the-new-retrieval-stack.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/note-gave-a-brown-bag-on-memory-migration-best-practices-kin.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/note-team-offsite-in-porto-next-month-kind-event-created-at.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/sprint-planning-on-monday-q3-memory-roadmap.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/event/upgraded-qdrant-to-1-12-across-all-environments.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/coffee-order-is-a-flat-white-with-oat-milk.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/deploys-to-kubernetes-on-aws-eks.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/has-a-cat-named-pixel.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/lives-in-lisbon-portugal.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/note-attends-the-lisbon-ai-meetup-monthly-kind-fact-created.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/note-maintains-an-open-source-memory-migration-tool-used-by.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/note-team-uses-mem0-for-long-term-agent-memory-in-production.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/note-the-agent-memory-store-hit-1m-embeddings-last-quarter-k.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/note-uses-vs-code-with-the-vim-extension-as-the-primary-edit.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/runs-a-homelab-with-a-nas-and-two-mini-pcs-for-self-hosted-s.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/team-ships-a-nightly-release-train-with-a-canary-stage.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/the-memory-service-stores-embeddings-in-a-qdrant-collection.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/the-okf-format-is-a-google-cloud-spec-for-portable-knowledge.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/uses-notion-for-personal-notes-and-docs.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/works-at-moorcheh-as-a-backend-engineer-on-the-memory-platfo.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/migrate-all-legacy-memory-stores-to-okf-by-end-of-quarter.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/note-publish-the-okf-adapter-showcase-to-the-community-kind.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/note-write-monthly-oss-blog-posts-documenting-the-migration.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/reduce-p95-retrieval-latency-under-120ms.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/run-a-10k-in-october.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/ship-v2-of-the-migration-cli-with-dry-run-previews.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/community-prs-for-adapters-tend-to-arrive-in-bursts-after-re.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/daily-summaries-are-more-useful-when-they-include-token-savi.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-retrieval-latency-spiked-after-the-1-12-upgrade-likely.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-the-canary-caught-a-regression-in-export-timestamps-las.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/users-of-the-migration-cli-often-ask-for-a-qdrant-source-ada.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/enjoys-dark-mode-uis-with-purple-accents.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/likes-llm-output-to-be-concise-hates-corporate-boilerplate.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-avoids-meetings-on-fridays-uses-them-for-deep-work-kind.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-dislikes-auto-generated-commit-messages-writes-manual-d.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-prefers-async-communication-over-real-time-chat-for-non.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-prefers-postgres-over-mysql-for-new-projects-kind-prefe.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-deep-work-blocks-before-noon-meetings-after-14-00-on.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-rfc-style-design-docs-before-large-refactors.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-small-focused-prs-under-300-lines.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-weekly-planning-on-monday-mornings-over-daily-standu.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/strong-preference-for-python-over-typescript-for-backend-ser.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/wants-dependency-upgrades-reviewed-separately-from-feature-w.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/wants-test-suites-to-fail-loudly-on-missing-ai-answers-rathe.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/relationship/collaborates-with-the-design-team-lead-sofia-on-ux-for-the-c.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/relationship/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/relationship/note-mentors-a-junior-engineer-named-rui-on-the-migration-to.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/relationship/note-pixel-the-cat-is-4-years-old-and-answers-to-pixel-only.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/relationship/reports-to-the-cto-marta.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/memories/relationship/works-closely-with-ana-ml-engineer-on-the-retrieval-pipeline.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/metrics/index.mdexamples/migrations/qdrant-to-okf/output/okf_bundle/metrics/overview.mdexamples/migrations/qdrant-to-okf/output/roundtrip_report.mdexamples/migrations/qdrant-to-okf/requirements.txtexamples/migrations/qdrant-to-okf/run_migration.pyexamples/migrations/qdrant-to-okf/seed_qdrant.pymemanto/cli/analyze/qdrant_export.pymemanto/cli/migrate/mappers.pymemanto/cli/migrate/runner.pysdks/typescript/README.mdsdks/typescript/src/index.tssdks/typescript/test/memanto.test.tstests/test_qdrant_migration.py
Comment on lines
+16
to
+22
| "payload": { | ||
| "metadata": { | ||
| "created_at": 1782534303780, | ||
| "memory_type": "preference", | ||
| "user_id": "tim@moorcheh.ai", | ||
| "agent_id": "assistant-v1", | ||
| "run_id": "run-5692" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== git status =="
git status --short
echo
echo "== changed files summary =="
git diff --stat || true
echo
echo "== locate relevant files =="
fd -a '^(export\.json(mapped)?|.*prefers.*\.md|.*created_at.*|.*mapped_preview\.jsonl)$' examples/migrations/qdrant-to-okf/output || true
echo
echo "== sample files =="
for f in examples/migrations/qdrant-to-okf/output/export.json \
examples/migrations/qdrant-to-okf/output/mapped_preview.jsonl \
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-rfc-style-design-docs-before-large-refactors.md \
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-small-focused-prs-under-300-lines.md \
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-weekly-planning-on-monday-mornings-over-daily-standu.md; do
if [ -f "$f" ]; then
echo "--- $f"
wc -l "$f"
sed -n '1,120p' "$f"
fi
done
echo
echo "== search for qdrant-to-okf scripts/seeds =="
fd -a 'qdrant|qdrant-to-okf|okf|memory|migration' examples scripts . 2>/dev/null | sed 's#^\./##' | head -200 || true
echo
echo "== exact identifiers in output =="
python3 - <<'PY'
from pathlib import Path
root = Path('examples/migrations/qdrant-to-okf/output')
for kind, suffix in [('', 'all')]:
matches = []
for p in root.rglob('*'):
if not p.is_file():
continue
try:
text = p.read_bytes()
except Exception:
continue
snippets = []
for target in ['tim@moorcheh.ai', 'assistant-v1', 'run-5692', 'run-4946', 'run-2799', 'run-2655']:
if target.encode() in text:
snippets.append(target)
if snippets:
matches.append((str(p), snippets))
print('\n'.join('{}: {}'.format(p, ', '.join(s)) for p,s in matches))
PYRepository: moorcheh-ai/memanto
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== git tracked files under examples/migrations/qdrant-to-okf =="
git ls-files examples/migrations/qdrant-to-okf | sed -n '1,200p'
echo
echo "== deterministic source-data scan for PII-like strings in all output files =="
python3 - <<'PY'
import json, re
from pathlib import Path
root = Path('examples/migrations/qdrant-to-okf/output')
hits = {'tim@moorcheh.ai': [], 'assistant-v1': [], 'run-': []}
p = re.compile(r'\brun-\d+\b')
for pth in sorted(root.rglob('*')):
if not pth.is_file():
continue
text = pth.read_bytes()
if b'tim@moorcheh.ai' in text:
hits['tim@moorcheh.ai'].append(str(pth))
if b'assistant-v1' in text:
hits['assistant-v1'].append(str(pth))
for m in p.finditer(text.decode(errors='ignore')):
target = m.group()
hits['run-'].append((str(pth), target))
break
print("tim@moorcheh.ai:")
for s in hits['tim@moorcheh.ai']:
print(" ", s)
print("assistant-v1:")
for s in hits['assistant-v1']:
print(" ", s)
print("first run-* per file:")
for item in hits['run-']:
print(" ", item[0], item[1])
print("export.json structure sample:")
data = json.loads((root / 'export.json').read_text())
print(json.dumps(data, indent=2, sort_keys=True)[:3000])
PYRepository: moorcheh-ai/memanto
Length of output: 20794
🌐 Web query:
docs.moorecheh tirm@moorcheh.ai
💡 Result:
Moorcheh (derived from the Persian word for "ant") is an information-theoretic AI search and memory platform designed for agentic AI applications [1][2]. It is developed to provide deterministic, index-free semantic search and retrieval infrastructure, aiming to replace traditional vector database stacks with a more efficient, serverless architecture [1][3]. Key aspects of the Moorcheh platform include: Technical Approach: Unlike traditional systems that rely on geometric distance as a proxy for semantic meaning, Moorcheh utilizes information theory to calculate semantic probability [2]. This approach is designed to provide explainable retrieval, high-fidelity storage, and stateful context for AI agents [1]. Deployment Options: Moorcheh is designed for flexibility across different environments [3]: - Cloud-Hosted: A serverless deployment option for rapid prototyping [3]. - Sovereign VPC: Allows deployment into a user's own AWS, GCP, or Azure environment, ensuring data remains within the private perimeter [4][3]. - On-Premise/Air-Gapped: Offers a source-available, self-hosted community edition for regulated or air-gapped environments [3]. - Edge: Optimized for running search and memory directly on edge devices [3]. Development Resources: - Documentation: Technical documentation, including API references and SDK guides, is available at https://docs.moorcheh.ai/ [1][5][6]. - SDKs: A Python SDK (moorcheh-sdk) is available for interacting with the platform's API [1][7]. - Console: Developers can manage API keys and monitor services via the Moorcheh Console at https://console.moorcheh.ai/ [1][8]. - GitHub: The project maintains public repositories, including the Python SDK and various integration examples, at https://github.com/moorcheh-ai [7]. The platform emphasizes security and cost-efficiency, marketing itself as a solution for health-tech, fintech, and other regulated industries that require compliant, performant data retrieval without the management overhead of traditional vector clusters [4][3].
Citations:
- 1: https://docs.moorcheh.ai/
- 2: https://moorcheh.ai/about
- 3: https://moorcheh.ai/products/overview
- 4: https://moorcheh.ai/
- 5: https://docs.moorcheh.ai/api-reference/introduction
- 6: https://docs.moorcheh.ai/python-sdk/introduction
- 7: https://github.com/moorcheh-ai
- 8: https://docs.moorcheh.ai/quickstart
Remove live user identifiers from committed fixtures.
The Qdrant seed uses tim@moorcheh.ai, and the migration outputs replay it in export.json, mapped_preview.jsonl, and every OKF footer. Replace the seed identity with a synthetic placeholder and regenerate all derived migration artifacts, including the listed preference files, the preview, and the full bundle.
📍 Affects 5 files
examples/migrations/qdrant-to-okf/output/export.json#L16-L22(this comment)examples/migrations/qdrant-to-okf/output/mapped_preview.jsonl#L1-L1examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-rfc-style-design-docs-before-large-refactors.md#L20-L25examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-small-focused-prs-under-300-lines.md#L20-L25examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-weekly-planning-on-monday-mornings-over-daily-standu.md#L20-L25
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/migrations/qdrant-to-okf/output/export.json` around lines 16 - 22,
Replace the live user identity in the Qdrant seed with a synthetic placeholder,
then regenerate all derived migration artifacts. Update
examples/migrations/qdrant-to-okf/output/export.json:16-22,
mapped_preview.jsonl:1-1, and the OKF footer metadata in each listed preference
file at lines 20-25; ensure the full okf_bundle is regenerated consistently and
contains no tim@moorcheh.ai identifier.
| - Collection: memories | ||
| - Qdrant point id: 52 | ||
| - Has vector: False | ||
| - Qdrant metadata: metadata={'created_at': 1785558303786, 'memory_type': 'commitment', 'user_id': 'tim@moorcheh.ai', 'agent_id': 'assistant-v1', 'run_id': 'run-7900'}; score=0.797; hash=68a430c9c8fa6401; categories=[... |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Redact identifiers from Qdrant supporting data before generating the bundle.
The mapper serializes the complete Qdrant payload into committed OKF documents. This exposes user and internal identifiers in generated output.
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/draft-the-okf-migration-adapter-pr-by-friday.md#L25-L25: Removeuser_id,agent_id, andrun_idbefore writing supporting data.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/uses-notion-for-personal-notes-and-docs.md#L25-L25: Removeuser_id,agent_id, andrun_idbefore writing supporting data.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/migrate-all-legacy-memory-stores-to-okf-by-end-of-quarter.md#L25-L25: Removeuser_id,agent_id, andrun_idbefore writing supporting data.
📍 Affects 3 files
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/draft-the-okf-migration-adapter-pr-by-friday.md#L25-L25(this comment)examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/uses-notion-for-personal-notes-and-docs.md#L25-L25examples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/migrate-all-legacy-memory-stores-to-okf-by-end-of-quarter.md#L25-L25
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/draft-the-okf-migration-adapter-pr-by-friday.md`
at line 25, Redact Qdrant supporting metadata before committing the generated
OKF documents by removing user_id, agent_id, and run_id from the serialized
payload. Apply this change to
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/draft-the-okf-migration-adapter-pr-by-friday.md:25,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/fact/uses-notion-for-personal-notes-and-docs.md:25,
and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/goal/migrate-all-legacy-memory-stores-to-okf-by-end-of-quarter.md:25.
Comment on lines
+3
to
+4
| title: 'note: Book flights for the Porto offsite. kind: commitment created_at: 178573...' | ||
| description: 'note: Book flights for the Porto offsite.' |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Keep raw note metadata out of migrated titles and content.
These records serialize kind and created_at into the title and body. The supporting-data footer already preserves unmapped fields. Update map_qdrant for this payload shape, regenerate the artifacts, and add a fixture assertion for title, content, and footer separation.
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md#L3-L4: use the note text as the title instead of appendingkindandcreated_at.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md#L17-L19: keep only the note text in the memory body.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-retrieval-latency-spiked-after-the-1-12-upgrade-likely.md#L3-L5: remove raw metadata from the title and description serialization.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-retrieval-latency-spiked-after-the-1-12-upgrade-likely.md#L19-L21: keep only the note text in the memory body.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-the-canary-caught-a-regression-in-export-timestamps-las.md#L3-L5: remove raw metadata from the title.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-the-canary-caught-a-regression-in-export-timestamps-las.md#L18-L20: keep only the note text in the memory body.
📍 Affects 3 files
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md#L3-L4(this comment)examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md#L17-L19examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-retrieval-latency-spiked-after-the-1-12-upgrade-likely.md#L3-L5examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-retrieval-latency-spiked-after-the-1-12-upgrade-likely.md#L19-L21examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-the-canary-caught-a-regression-in-export-timestamps-las.md#L3-L5examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-the-canary-caught-a-regression-in-export-timestamps-las.md#L18-L20
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md`
around lines 3 - 4, Update map_qdrant for note payloads so titles and memory
bodies contain only the note text, while kind and created_at remain exclusively
in the supporting-data footer; regenerate the artifacts and add fixture
assertions covering title, content, and footer separation. Apply this to
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/commitment/note-book-flights-for-the-porto-offsite-kind-commitment-crea.md
at lines 3-4 and 17-19,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-retrieval-latency-spiked-after-the-1-12-upgrade-likely.md
at lines 3-5 and 19-21, and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/observation/note-the-canary-caught-a-regression-in-export-timestamps-las.md
at lines 3-5 and 18-20.
| - Collection: memories | ||
| - Qdrant point id: 46 | ||
| - Has vector: False | ||
| - Qdrant metadata: metadata={'created_at': 1782707103786, 'memory_type': 'decision', 'user_id': 'tim@moorcheh.ai', 'agent_id': 'assistant-v1', 'run_id': 'run-2887'}; score=0.858; hash=d05b37f6d567f530; categories=['w... |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Remove user identifiers from committed migration artifacts.
The supporting-data footer commits user_id: 'tim@moorcheh.ai'. These files are part of the example output bundle. Redact user identifiers or allowlist non-sensitive metadata before generating the artifacts.
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/chose-qdrant-over-pinecone-for-the-embedding-store-self-host.md#L26-L26: Remove or redactuser_id.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/decided-the-migration-cli-must-never-silently-drop-unmapped.md#L25-L25: Remove or redactuser_id.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/likes-llm-output-to-be-concise-hates-corporate-boilerplate.md#L25-L25: Remove or redactuser_id.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-deep-work-blocks-before-noon-meetings-after-14-00-on.md#L25-L25: Remove or redactuser_id.
📍 Affects 4 files
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/chose-qdrant-over-pinecone-for-the-embedding-store-self-host.md#L26-L26(this comment)examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/decided-the-migration-cli-must-never-silently-drop-unmapped.md#L25-L25examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/likes-llm-output-to-be-concise-hates-corporate-boilerplate.md#L25-L25examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-deep-work-blocks-before-noon-meetings-after-14-00-on.md#L25-L25
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/chose-qdrant-over-pinecone-for-the-embedding-store-self-host.md`
at line 26, Remove or redact the user_id field from the supporting-data footers
in all four committed migration artifacts:
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/chose-qdrant-over-pinecone-for-the-embedding-store-self-host.md:26-26,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/decided-the-migration-cli-must-never-silently-drop-unmapped.md:25-25,
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/likes-llm-output-to-be-concise-hates-corporate-boilerplate.md:25-25,
and
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/prefers-deep-work-blocks-before-noon-meetings-after-14-00-on.md:25-25.
Preserve non-sensitive metadata while ensuring no user identifier is committed.
Comment on lines
+18
to
+20
| note: Moved CI to GitHub Actions with a 5-min warm cache. | ||
| kind: decision | ||
| created_at: 1785385503786 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Extract raw note values before generating memory content.
These memories contain kind: and numeric created_at: fields as user content. The fields should remain typed metadata or supporting data. Keeping them in the body pollutes search content and causes the generated titles to include backend metadata.
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-moved-ci-to-github-actions-with-a-5-min-warm-cache-kind.md#L18-L20: Map only the note text into memory content.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-standardized-on-uv-for-python-dependency-management-kin.md#L18-L20: Map only the note text into memory content.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-avoids-meetings-on-fridays-uses-them-for-deep-work-kind.md#L18-L20: Map only the note text into memory content.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-dislikes-auto-generated-commit-messages-writes-manual-d.md#L18-L20: Map only the note text into memory content.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-prefers-async-communication-over-real-time-chat-for-non.md#L19-L21: Map only the note text into memory content.examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-prefers-postgres-over-mysql-for-new-projects-kind-prefe.md#L17-L19: Map only the note text into memory content.
📍 Affects 6 files
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-moved-ci-to-github-actions-with-a-5-min-warm-cache-kind.md#L18-L20(this comment)examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-standardized-on-uv-for-python-dependency-management-kin.md#L18-L20examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-avoids-meetings-on-fridays-uses-them-for-deep-work-kind.md#L18-L20examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-dislikes-auto-generated-commit-messages-writes-manual-d.md#L18-L20examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-prefers-async-communication-over-real-time-chat-for-non.md#L19-L21examples/migrations/qdrant-to-okf/output/okf_bundle/memories/preference/note-prefers-postgres-over-mysql-for-new-projects-kind-prefe.md#L17-L19
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-moved-ci-to-github-actions-with-a-5-min-warm-cache-kind.md`
around lines 18 - 20, Update the memory-content mapping so only each raw note
value is included in generated content, while kind and numeric created_at remain
typed metadata or supporting data:
examples/migrations/qdrant-to-okf/output/okf_bundle/memories/decision/note-moved-ci-to-github-actions-with-a-5-min-warm-cache-kind.md:18-20,
note-standardized-on-uv-for-python-dependency-management-kin.md:18-20,
preference/note-avoids-meetings-on-fridays-uses-them-for-deep-work-kind.md:18-20,
note-dislikes-auto-generated-commit-messages-writes-manual-d.md:18-20,
note-prefers-async-communication-over-real-time-chat-for-non.md:19-21, and
note-prefers-postgres-over-mysql-for-new-projects-kind-prefe.md:17-19. Apply the
same raw-note-only change at every listed site.
Comment on lines
+91
to
+93
| Anything that doesn't map onto the schema is packed into a bounded | ||
| `[Supporting data]` markdown footer on the memory — searchable, visible, and | ||
| lossless on re-import (unmapped fields are preserved, nothing is dropped). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Do not describe the bounded footer as lossless.
_format_supporting_data truncates individual values and caps the full footer. The generated bundle also contains truncated metadata. Unmapped fields can therefore be lost during migration.
State the limits here. If lossless retention is required, persist the raw payload in a separate artifact or attachment.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/migrations/qdrant-to-okf/README.md` around lines 91 - 93, Update the
migration README text around the [Supporting data] footer to remove the
“lossless” claim and explicitly state that individual values and the complete
footer are truncated or capped, so unmapped fields may be lost. If lossless
retention is required, direct users to persist the raw payload as a separate
artifact or attachment.
| # Everything else rides along for the mapper's supporting-data footer. | ||
| "payload": {k: v for k, v in flat.items() if k not in _TEXT_KEYS}, | ||
| "collection": collection, | ||
| "has_vector": bool(point.vector), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python -m pip install --quiet 'qdrant-client>=1.9,<2'
python - <<'PY'
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, PointStruct, VectorParams
client = QdrantClient(":memory:")
client.create_collection(
collection_name="memories",
vectors_config=VectorParams(size=2, distance=Distance.COSINE),
)
client.upsert(
collection_name="memories",
points=[PointStruct(id=1, vector=[0.1, 0.2], payload={"text": "memory"})],
)
without_vectors, _ = client.scroll(
collection_name="memories", with_payload=True, with_vectors=False
)
with_vectors, _ = client.scroll(
collection_name="memories", with_payload=True, with_vectors=True
)
assert not without_vectors[0].vector
assert with_vectors[0].vector
PYRepository: moorcheh-ai/memanto
Length of output: 194
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Repo files around qdrant_export.py:\\n'
fd -a 'qdrant_export.py|qdrant' . | sed 's#^\./##' | head -50
printf '\\nqdrant_export.py excerpt:\\n'
if [ -f memanto/cli/analyze/qdrant_export.py ]; then
nl -ba memanto/cli/analyze/qdrant_export.py | sed -n '100,155p'
fi
printf '\\nSearch for dump_collection/from_point and has_vector usages:\\n'
rg -n "dump_collection|point_to_memory|has_vector|with_vectors|QdrantClient|with_payload" -S .Repository: moorcheh-ai/memanto
Length of output: 1085
🌐 Web query:
qdrant-client python scroll with_vectors parameter Point vector unavailable
💡 Result:
In the Qdrant Python client, the scroll method uses the parameter with_vectors (plural), not with_vector (singular), to control whether stored vectors are returned in the result [1][2]. If you are receiving vector=None or an unavailable vector despite setting a flag, consider the following common causes: 1. Parameter Name: Ensure you are using with_vectors=True [1][2]. Using with_vector (singular) is a common mistake that may not trigger an error but will fail to retrieve the vector data [2]. 2. Default Behavior: The default value for with_vectors is False [1]. If the parameter is omitted, no vectors will be returned [1]. 3. Vector Normalization: If the returned vector values appear different from those you uploaded, note that Qdrant automatically normalizes vectors if the collection is configured with cosine distance [3][4]. This is expected behavior to ensure consistent similarity calculations [4]. 4. Multi-vector or Named Vectors: If you are using named vectors or multi-vector configurations, ensure your collection and points are correctly defined [3]. If you update vectors in local mode, there have been reported issues where vectors may not appear as expected; switching to a standard Docker-based deployment can resolve environment-specific behavior [5]. 5. Server/Environment Issues: If you encounter crashes (e.g., panics) or errors when requesting vectors, this can sometimes stem from incompatible storage configurations, such as running Qdrant via WSL with bind mounts on Windows, which can interfere with the memory-mapped files Qdrant relies on [6]. If you continue to experience missing vectors, verify your collection configuration using get_collection to ensure the vector parameters are correctly set and that data was successfully persisted [7].
Citations:
- 1: https://github.com/qdrant/qdrant-client/blob/cd5eb259/qdrant_client/qdrant_client.py
- 2: [BUG] Qdrant python client doesn't return vector even with with_vector option set qdrant/qdrant-client#189
- 3: Empty vectors while scrolling unnamed vector collection qdrant/qdrant-client#434
- 4: Uploaded and retrived vectors are different qdrant/qdrant-client#727
- 5: MultiVector Cannot be updated separately qdrant/qdrant-client#1029
- 6: scroll API with with_vectors=True, Qdrant crashes with a panic qdrant/qdrant#7076
- 7: https://github.com/qdrant/qdrant-client
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'qdrant_export.py excerpt:\\n'
awk '{printf "%6d %s\n", NR, $0}' memanto/cli/analyze/qdrant_export.py | sed -n '100,155p'
printf '\\nseed_qdrant.py excerpt:\\n'
awk '{printf "%6d %s\n", NR, $0}' examples/migrations/qdrant-to-okf/seed_qdrant.py | sed -n '1,220p'
printf '\\nqdrant usages excerpt around exports/tests:\\n'
awk '{printf "%6d %s\n", NR, $0}' tests/test_qdrant_migration.py | sed -n '1,260p'Repository: moorcheh-ai/memanto
Length of output: 22422
Preserve vector metadata when exporting Qdrant points.
dump_collection() scrolls with with_vectors=False, so point.vector is unavailable and has_vector is False for vector-bearing points. Request vectors in this scroll operation, compute has_vector, then discard the vector payload. Add memories[0]["has_vector"] to the embedded-Qdrant exporter test so this regression has a guard.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@memanto/cli/analyze/qdrant_export.py` at line 129, Update dump_collection()
to request vectors in its Qdrant scroll operation, compute has_vector from
point.vector, and discard the vector payload before exporting. Extend the
embedded-Qdrant exporter test to assert memories[0]["has_vector"] is true for a
vector-bearing point.
Comment on lines
+173
to
+183
| client = QdrantClient(":memory:") if args.in_memory else QdrantClient( | ||
| url=args.url, api_key=args.api_key | ||
| ) | ||
| if args.in_memory: | ||
| # The seed script stores its client handle here so the dump shares it. | ||
| from qdrant_client import QdrantClient as _QC | ||
| from qdrant_client.local.qdrant_local import QdrantLocal | ||
|
|
||
| _SHARED = getattr(sys.modules.get("__main__"), "_QD_SHARED", None) | ||
| if _SHARED is not None and isinstance(_SHARED, QdrantLocal): | ||
| client._client = _SHARED # reuse the seeded in-memory store |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python -m pip install --quiet 'qdrant-client>=1.9,<2'
python - <<'PY'
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
first = QdrantClient(":memory:")
first.create_collection(
collection_name="memories",
vectors_config=VectorParams(size=2, distance=Distance.COSINE),
)
second = QdrantClient(":memory:")
names = [collection.name for collection in second.get_collections().collections]
assert "memories" not in names, names
PYRepository: moorcheh-ai/memanto
Length of output: 194
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | sed -n '1,120p'
echo "== locate qdrant_export.py =="
fd -a 'qdrant_export\.py$' . || true
echo "== outline qdrant_export if found =="
for f in $(fd 'qdrant_export\.py$' .); do
echo "--- $f ---"
wc -l "$f"
ast-grep outline "$f" || true
echo "== relevant lines =="
sed -n '130,215p' "$f" | cat -n
done
echo "== search _QD_SHARED and in_memory args =="
rg -n "_QD_SHARED|in_memory|--in-memory|QdrantClient\\(\\\":memory\"" .Repository: moorcheh-ai/memanto
Length of output: 12363
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== test around qdrant export in-memory =="
sed -n '90,150p' tests/test_qdrant_migration.py | cat -n
echo "== any seed/demo scripts that mention _QD_SHARED or in-memory =="
rg -n --iglob '*_seed*' --iglob '*seed*' --iglob '*demo*' --iglob '*qdrant*' "_QD_SHARED|QdrantClient\\(\"\\:memory\\\"\"|--in-memory|in_memory|in-memory" .Repository: moorcheh-ai/memanto
Length of output: 4282
🌐 Web query:
qdrant-client Python QdrantClient ":memory:" local storage behavior
💡 Result:
When you initialize the QdrantClient with location=":memory:", the client operates in a volatile, in-memory mode [1][2]. This configuration is specifically designed for development, prototyping, testing, and CI/CD pipelines, as it does not persist any data to the disk [1][3]. Key characteristics of this behavior include: Data Volatility: All collections and points are stored exclusively in RAM [1][2]. As soon as the Python process terminates or the QdrantClient instance is closed or garbage collected, all stored data is permanently lost [2][4]. Persistence: There is no disk persistence mechanism enabled when using :memory:. If you require data to persist across script executions or restarts, you must use the path parameter instead (e.g., QdrantClient(path="path/to/db")), which directs the client to save data in a local directory [1][3]. Limitations: The in-memory mode is not recommended for production environments or for managing large datasets [2][4]. It is intended for lightweight experiments and simple testing [1][5]. Some advanced storage configurations or features may have limited or unimplemented functionality in this mode compared to a full Qdrant server instance [5]. In summary, use :memory: when you need a clean, ephemeral state for every run, and use path when you need your data to survive the lifecycle of your Python application [1][3].
Citations:
- 1: https://github.com/qdrant/qdrant-client?tab=readme-ov-file
- 2: https://github.com/qdrant/qdrant-client/blob/cd5eb259/qdrant_client/local/qdrant_local.py
- 3: https://github.com/qdrant/qdrant-client
- 4: https://python-client.qdrant.tech/_modules/qdrant_client/local/async_qdrant_local
- 5: Setting Parameters in Qdrant Client running in " Memory" Issue: qdrant/qdrant-client#683
Remove the misleading --in-memory CLI mode.
Each QdrantClient(":memory:") owns its own ephemeral in-memory store, so a dump process cannot see collections created by a separate seed process. examples/migrations/qdrant-to-okf/seed_qdrant.py also never sets __main__._QD_SHARED, so the branch creates an empty store and reports the collection as missing. Use a shared persistent local path for cross-process dumps, or remove --in-memory and require the seed and dump to run in the same process.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@memanto/cli/analyze/qdrant_export.py` around lines 173 - 183, Remove the
--in-memory CLI mode and its QdrantClient(":memory:") and _QD_SHARED reuse
branch in the export flow. Require the dump to use the persistent local storage
path shared with the seed process, updating argument handling and client
initialization accordingly so cross-process exports access the seeded
collections.
Comment on lines
+100
to
+114
| def test_map_qdrant_preserves_timestamp_via_ms_epoch(): | ||
| """Millisecond epoch timestamps surface as real datetimes.""" | ||
| export = _export( | ||
| [ | ||
| { | ||
| "id": "4", | ||
| "content": "Seeded memory with ms epoch.", | ||
| "type": "fact", | ||
| "tags": [], | ||
| "created_at": "2026-07-01T00:00:00+00:00", | ||
| } | ||
| ] | ||
| ) | ||
| row = map_qdrant(export)[0] | ||
| assert row["created_at"] == datetime(2026, 7, 1, tzinfo=timezone.utc) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Test an actual millisecond epoch.
The fixture supplies an ISO-8601 string, so this test does not exercise millisecond conversion. The embedded-Qdrant test supplies milliseconds but does not assert the mapped timestamp. A regression in _as_utc_iso can pass both tests.
Use int(expected.timestamp() * 1000) as created_at, then assert the expected UTC datetime.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_qdrant_migration.py` around lines 100 - 114, Update
test_map_qdrant_preserves_timestamp_via_ms_epoch to set the fixture’s created_at
to int(expected.timestamp() * 1000) instead of an ISO-8601 string, where
expected is the target UTC datetime, and assert the mapped row’s created_at
equals expected.
…i#1609) 23s terminal demo of the full freedom loop: seed -> dump -> map -> OKF bundle -> round-trip QA. Rendered from real run_migration.py output.
…oorcheh-ai#1609) Address coderabbit review: - run_migration.py: exit with clear error if map_qdrant returns 0 rows instead of IndexError on rows[0] - README: bounded supporting-data footer preserves unmapped fields but is not strictly lossless at the cap — wording now matches reality
Address coderabbit review (Data Integrity, Major): the round-trip check previously validated only row counts + substring presence. Now every source record is tracked by source_ref through the OKF bundle and back: - missing/extra record detection (source_ref counter difference) - content-continuity check (source body must survive re-import) - type re-classification reported as informational (OKF types are free-form) - parity table added to roundtrip_report.md Verified: 61/61 records round-trip, record parity PASS, golden QA 5/5.
Author
|
Addressed the coderabbit review — round-trip validation upgraded from row-count + substring checks to record-level parity (commit 98d315c). Every source record is now tracked by
Verified locally: 61/61 records round-trip, record parity PASS, golden QA 5/5 (100%). Full table in Also fixed in this PR: empty-mapping guard before |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Qdrant → Memanto → OKF: Path B Migration Adapter (Great Memory Migration #1609)
First-class escape route for the most common production vector store backing agent memory (Mem0-on-Qdrant, LangChain vectorstores, RAG stacks): a raw Qdrant collection → Memanto → portable OKF markdown.
What's included
memanto/cli/analyze/qdrant_export.py(new) — scrolls a Qdrant collection into the provider-style export JSON consumed bymemanto migrate --filemap_qdrantinmemanto/cli/migrate/mappers.py— slots Qdrant payload fields onto the Memanto schema; unmapped fields ride in a bounded[Supporting data]footer, preserved through the OKF round tripexamples/migrations/qdrant-to-okf/— lived-in seed data (80 points, 6 weeks of memory, Mem0/LangChain/raw payload shapes generated by a real embedded Qdrant run), one-command runnable loop, mapping table, savings report, OKF bundle outputMigration summary
Payload conventions handled
text+metadata(Mem0-on-Qdrant),page_content+metadata(LangChain), bare attribute dicts (RAG chunks). Full mapping table inexamples/migrations/qdrant-to-okf/README.md.Quick start (~30 seconds, zero infra)
Artifacts:
export.json,mapped_preview.jsonl,okf_bundle/,roundtrip_report.md(5/5 golden QA).Sample OKF bundle
See
examples/migrations/qdrant-to-okf/output/okf_bundle/—index.md+memories/<type>/<slug>.mdwith YAML frontmatter.⏱️ ### Demo video
Qdrant → OKF migration demo (MP4)
23-second run of the full freedom loop — seed → dump → map → OKF bundle → round-trip QA (61/61 re-imported, 5/5 golden recall), rendered from real
run_migration.pyoutput.Social posts coming (posting to X/LinkedIn shortly). Ref: #1609
Summary by CodeRabbit
New Features
Documentation