feat(ops): backup / restore / DR tooling with a CI zero-data-loss gate

PCMI keeps all durable state in PostgreSQL but shipped no backup strategy or
tooling — an undefendable "production-grade" claim. Adds:

- scripts/backup/pcmi_backup.sh  — pg_dump → compressed, timestamped custom-format
  archive (prints the path).
- scripts/backup/pcmi_restore.sh — pg_restore into DATABASE_URL; REFUSES a
  non-empty target unless FORCE=1, with --clean --if-exists --exit-on-error.
- scripts/backup/ci_backup_restore_test.sh — seed → backup → wipe → restore →
  assert (tenant/table counts + a marker row survive; and that restore refuses
  to clobber without FORCE).
- Makefile: backup, restore, backup-restore-test.
- docs/runbooks/backup-restore.md — scheduling, restore, verification, DR
  order-of-operations, PITR guidance, and a pg client/server version note.
- CI job `backup-restore` (pg16 service) runs the full cycle on every PR, so a
  regression that breaks restore fails the build.

Verified locally: full round-trip with matching pg16 tools preserves every row
across backup → schema wipe → restore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: marco-spagn

.github/workflows/ci.yml

@@ -11,6 +11,7 @@
 #   integration-e2e    → needs [go, golangci-lint]  (skips if no OPENAI_API_KEY; no retry)
 #   integration-graph  → needs [go, golangci-lint]  (complete cognitive graph matrix gate)
 #   retrieval-eval     → needs [go, golangci-lint]  (skips if no OPENAI_API_KEY; gates recall@k/nDCG on eval/retrieval/thresholds.json)
+#   backup-restore     → needs [go, golangci-lint]  (seed→backup→wipe→restore→assert zero data loss)
 # CodeQL: separate workflow (.github/workflows/codeql.yml)
 # Local parity: make ci-like-github → scripts/ci_like_github.sh (phases A–G)
 # Version for smoke: scripts/ci/resolve_version.sh (internal/version/version.go)
@@ -655,6 +656,38 @@ jobs:
         if: always() && steps.openai.outputs.enabled == 'true'
         run: docker compose down -v --remove-orphans
 
+  backup-restore:
+    name: Backup / restore (zero data loss)
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    needs: [go, golangci-lint]
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        env:
+          POSTGRES_DB: pcmi
+          POSTGRES_USER: pcmi
+          POSTGRES_PASSWORD: pcmi
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U pcmi -d pcmi"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+    steps:
+      - uses: actions/checkout@v7
+
+      - name: Install postgresql-client
+        run: sudo apt-get update && sudo apt-get install -y postgresql-client
+
+      - uses: ./.github/actions/pcmi-postgres-migrate
+
+      - name: Seed → backup → wipe → restore → assert
+        env:
+          DATABASE_URL: postgres://pcmi:pcmi@127.0.0.1:5432/pcmi?sslmode=disable
+        run: bash scripts/backup/ci_backup_restore_test.sh
+
   integration-graph:
     name: Integration graph (complete AGE matrix)
     runs-on: ubuntu-latest

CHANGELOG.md

@@ -11,6 +11,8 @@ the public API version exposed by `/v1/version` and the gRPC `Version` RPC.
 
 ### Added
 
+- **Backup / restore / DR tooling** (`scripts/backup/`): `pcmi_backup.sh` wraps `pg_dump` into a compressed, timestamped custom-format archive; `pcmi_restore.sh` wraps `pg_restore` and **refuses to overwrite a populated database unless `FORCE=1`**. Makefile targets `backup`, `restore`, `backup-restore-test`. A `docs/runbooks/backup-restore.md` runbook covers scheduled backups, restore, backup verification, DR order-of-operations, and PITR guidance. A CI job (`backup-restore`) runs the full **seed → backup → wipe → restore → assert** cycle on every PR so a regression that breaks restore fails the build. Redis holds only transient state and is not backed up.
+
 - **Entity extraction Phase A**: migration `022_extraction_profiles.sql`; `EXTRACTION_ENABLED` worker/API flag; tenant profiles (`GET/PUT/DELETE /v1/extraction-profiles/{id}`); LLM slot extraction into `metadata.pcmi_extract` (`GET/POST /v1/memories/extraction/{memory_id}`). See [cognitive-graph-entities.md](docs/cognitive-graph-entities.md).
 
 - **Entity extraction Phase B**: migration `023_entity_graph.sql`; promoted slots become `:Entity` vertices with `:mentions` edges from `:Memory` when extraction validates and AGE is available; `GET /v1/graph/entities/memory`, `GET /v1/graph/entities/related` (by `kind`+`key` or shared entities via `memory_id`). See [cognitive-graph-entities.md](docs/cognitive-graph-entities.md).

Makefile

@@ -6,7 +6,7 @@
         changelog-unreleased changelog-tag tag-release examples-smoke-structural examples-smoke \
         helm-lint helm-template helm-package admin-list-keys bench quickstart graph-ui graph-ui-entities demo test-cognitive-graph test-cognitive-graph-matrix \
         graph-realistic-generate graph-realistic-validate graph-realistic-smoke graph-realistic-audit graph-soc-loader-test demo_2 demo-cti-operational demo_soc demo_cti \
-        eval-retrieval eval-retrieval-validate
+        eval-retrieval eval-retrieval-validate backup restore backup-restore-test
 
 GOLANGCI_LINT_VERSION ?= v2.12.2
 GRPC_HOST ?= localhost:50051
@@ -127,6 +127,22 @@ smoke-dedup:
 admin-list-keys:
 	DATABASE_URL=$(DATABASE_URL) go run ./cmd/pcmi-admin list
 
+# ── Backup / restore / DR (docs/runbooks/backup-restore.md) ───────────────────
+# make backup                       → ./backups/pcmi-<ts>.dump
+# make backup BACKUP_DIR=/mnt/bk
+# make restore BACKUP_FILE=./backups/pcmi-<ts>.dump [FORCE=1]
+BACKUP_DIR ?= ./backups
+backup:
+	@DATABASE_URL="$(DATABASE_URL)" bash scripts/backup/pcmi_backup.sh "$(BACKUP_DIR)"
+
+restore:
+	@test -n "$(BACKUP_FILE)" || (echo "usage: make restore BACKUP_FILE=<archive> [FORCE=1]" && exit 2)
+	@DATABASE_URL="$(DATABASE_URL)" FORCE="$(FORCE)" bash scripts/backup/pcmi_restore.sh "$(BACKUP_FILE)"
+
+# End-to-end: seed → backup → wipe → restore → assert (destructive; test DB only).
+backup-restore-test:
+	@DATABASE_URL="$(DATABASE_URL)" bash scripts/backup/ci_backup_restore_test.sh
+
 # Shortcuts
 up: infra-up
 down: infra-down

docs/runbooks/backup-restore.md

@@ -0,0 +1,102 @@
+# Runbook — Backup & Restore / Disaster Recovery
+
+PCMI keeps all durable state in **PostgreSQL** (memory entries and their
+versions, links, sessions, distilled knowledge, entity graph tables, tenants,
+API keys, audit log). Redis holds only the transient event stream and rate-limit
+counters — it is **not** a source of truth and does not need backing up.
+
+This runbook covers logical backups with `pg_dump` and restore with
+`pg_restore`. For very large or low-RPO deployments, prefer your platform's
+**physical** backup / point-in-time recovery (PITR) — see *Beyond logical dumps*.
+
+## Tooling
+
+Two wrappers under `scripts/backup/`:
+
+| Script | Purpose |
+|---|---|
+| `pcmi_backup.sh [OUT_DIR]` | `pg_dump` → compressed, timestamped custom-format archive. Prints the archive path. |
+| `pcmi_restore.sh <archive>` | `pg_restore` into `DATABASE_URL`. **Refuses a non-empty target unless `FORCE=1`.** |
+
+Both read the target from `DATABASE_URL`. Makefile shortcuts:
+
+```bash
+make backup                         # → ./backups/pcmi-<ts>.dump
+make backup BACKUP_DIR=/mnt/backups
+make restore BACKUP_FILE=./backups/pcmi-20260101T000000Z.dump
+make restore BACKUP_FILE=... FORCE=1   # overwrite a populated DB
+```
+
+> **Version note:** run `pg_dump` with a client version **equal to or newer than
+> the server**, and restore with a client **matching the target server major
+> version**. A newer client dumping for an older server can emit settings the
+> older server rejects (e.g. `transaction_timeout`, added in PG 17). PCMI targets
+> **PostgreSQL 16**; use `postgresql-client-16`. The dockerized DB already ships
+> matching tools — `docker exec pcmi-postgres pg_dump …` always version-matches.
+
+## Take a backup
+
+```bash
+export DATABASE_URL='postgres://pcmi:pcmi@db-host:5432/pcmi?sslmode=disable'
+./scripts/backup/pcmi_backup.sh /mnt/backups
+# → /mnt/backups/pcmi-20260724T093000Z.dump
+```
+
+Store the archive off-box (object storage, another region). Automate with a cron
+/ CronJob calling the same script; keep N daily + M weekly copies.
+
+## Restore
+
+Into an **empty** database (fresh DR target):
+
+```bash
+export DATABASE_URL='postgres://pcmi:pcmi@dr-host:5432/pcmi?sslmode=disable'
+createdb -h dr-host -U pcmi pcmi          # if the database does not exist yet
+./scripts/backup/pcmi_restore.sh /mnt/backups/pcmi-20260724T093000Z.dump
+```
+
+Over an **existing** database (accepts data loss — the current contents are
+dropped and replaced):
+
+```bash
+FORCE=1 ./scripts/backup/pcmi_restore.sh /mnt/backups/pcmi-20260724T093000Z.dump
+```
+
+The restore uses `pg_restore --clean --if-exists --exit-on-error`, so a partial
+or corrupt archive fails loudly instead of leaving a half-restored database.
+
+## Verify a backup (do this regularly — an untested backup is not a backup)
+
+Restore into a throwaway database and check row counts:
+
+```bash
+createdb -h localhost -U pcmi pcmi_verify
+DATABASE_URL='postgres://pcmi:pcmi@localhost:5432/pcmi_verify?sslmode=disable' \
+  ./scripts/backup/pcmi_restore.sh <archive>
+psql "$DATABASE_URL" -c "SELECT count(*) FROM memory_entries;"
+dropdb -h localhost -U pcmi pcmi_verify
+```
+
+CI runs the full **seed → backup → wipe → restore → assert** cycle on every PR
+(`scripts/backup/ci_backup_restore_test.sh`, the `backup-restore` job), so a
+regression that breaks restore fails the build.
+
+## Disaster recovery — order of operations
+
+1. Provision a PostgreSQL 16 instance (with the `vector`, `ltree`, `pg_trgm`
+   extensions available; the dump recreates them).
+2. Restore the most recent verified archive (see *Restore*, empty target).
+3. Point `DATABASE_URL` (and `DATABASE_READ_URL`, if used) at the new instance.
+4. Start API + worker. Redis can be empty — the worker rebuilds its consumer
+   group; embeddings already persisted are restored with the dump.
+5. Smoke: `curl $API/v1/ready` → `database_ok:true`, then a `POST /v1/retrieve`.
+
+**RPO/RTO:** with periodic logical dumps, RPO = your backup interval and RTO =
+restore time (minutes for small/medium corpora). For tighter objectives use PITR.
+
+## Beyond logical dumps (PITR)
+
+For large corpora or near-zero RPO, use physical backups + WAL archiving
+(`pg_basebackup` + `archive_command`, or a managed service's continuous backup —
+RDS/Cloud SQL/Crunchy). PITR replays WAL to any point in time; logical dumps
+remain useful for portable, cross-version, per-database exports.

scripts/backup/ci_backup_restore_test.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# End-to-end proof that a PCMI backup can be restored with zero data loss:
+#   seed a marker → backup → wipe the schema → restore → assert the marker and
+#   row counts survived. Exits non-zero on any mismatch (CI gate).
+#
+# Requires: DATABASE_URL pointing at a migrated PCMI database + postgresql-client.
+set -euo pipefail
+
+DATABASE_URL="${DATABASE_URL:?DATABASE_URL is required}"
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+psql_q() { psql "$DATABASE_URL" -tAc "$1"; }
+
+marker="backup-restore-$(date +%s)"
+
+echo "== 1. seed a marker tenant =="
+psql_q "INSERT INTO tenants (slug, name) VALUES ('$marker', '$marker')" >/dev/null
+tenants_before="$(psql_q "SELECT count(*) FROM tenants")"
+tables_before="$(psql_q "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'")"
+echo "   tenants=$tenants_before tables=$tables_before"
+
+echo "== 2. backup =="
+backup="$(DATABASE_URL="$DATABASE_URL" bash "$HERE/pcmi_backup.sh" "$WORK")"
+[ -f "$backup" ] || { echo "FAIL: backup file missing"; exit 1; }
+
+echo "== 3. wipe the schema =="
+psql "$DATABASE_URL" -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;" >/dev/null
+wiped="$(psql_q "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'")"
+[ "$wiped" = "0" ] || { echo "FAIL: schema not empty after wipe ($wiped tables)"; exit 1; }
+echo "   confirmed empty"
+
+echo "== 4. restore =="
+DATABASE_URL="$DATABASE_URL" bash "$HERE/pcmi_restore.sh" "$backup"
+
+echo "== 5. assert no data loss =="
+tenants_after="$(psql_q "SELECT count(*) FROM tenants")"
+tables_after="$(psql_q "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'")"
+found="$(psql_q "SELECT count(*) FROM tenants WHERE slug='$marker'")"
+echo "   tenants=$tenants_after tables=$tables_after marker_found=$found"
+
+fail=0
+[ "$tenants_after" = "$tenants_before" ] || { echo "FAIL: tenant count $tenants_before → $tenants_after"; fail=1; }
+[ "$tables_after" = "$tables_before" ]   || { echo "FAIL: table count $tables_before → $tables_after"; fail=1; }
+[ "$found" = "1" ]                       || { echo "FAIL: marker tenant not restored"; fail=1; }
+
+# The restore safety gate must refuse a second restore over the now-populated DB.
+echo "== 6. assert restore refuses to clobber without FORCE =="
+if DATABASE_URL="$DATABASE_URL" bash "$HERE/pcmi_restore.sh" "$backup" >/dev/null 2>&1; then
+  echo "FAIL: restore overwrote a populated DB without FORCE=1"; fail=1
+else
+  echo "   refused (as expected)"
+fi
+
+if [ "$fail" != "0" ]; then
+  echo "BACKUP/RESTORE TEST: FAILED"; exit 1
+fi
+echo "BACKUP/RESTORE TEST: PASSED — zero data loss across backup → wipe → restore"

scripts/backup/pcmi_backup.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+# PCMI backup — pg_dump wrapper producing a compressed, timestamped custom-format
+# archive suitable for pcmi_restore.sh / pg_restore.
+#
+# Usage:
+#   DATABASE_URL=postgres://user:pass@host:5432/pcmi ./scripts/backup/pcmi_backup.sh [OUT_DIR]
+#
+# Env:
+#   DATABASE_URL       required — libpq connection URI
+#   PCMI_BACKUP_DIR    default output dir when OUT_DIR arg is omitted (default ./backups)
+#
+# Prints the archive path on stdout (so callers can capture it); progress on stderr.
+set -euo pipefail
+
+DATABASE_URL="${DATABASE_URL:-}"
+OUT_DIR="${1:-${PCMI_BACKUP_DIR:-./backups}}"
+
+if [ -z "$DATABASE_URL" ]; then
+  echo "[backup] DATABASE_URL is required" >&2
+  exit 2
+fi
+if ! command -v pg_dump >/dev/null 2>&1; then
+  echo "[backup] pg_dump not found — install postgresql-client" >&2
+  exit 2
+fi
+
+mkdir -p "$OUT_DIR"
+ts="$(date -u +%Y%m%dT%H%M%SZ)"
+out="$OUT_DIR/pcmi-${ts}.dump"
+
+echo "[backup] pg_dump → $out" >&2
+# --format=custom: compressed, restorable with pg_restore (selective/parallel).
+# --no-owner/--no-privileges: portable across roles (restore into any owner).
+pg_dump "$DATABASE_URL" \
+  --format=custom \
+  --no-owner \
+  --no-privileges \
+  --file="$out"
+
+size="$(du -h "$out" | cut -f1)"
+echo "[backup] wrote ${size} — $out" >&2
+echo "$out"

scripts/backup/pcmi_restore.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# PCMI restore — pg_restore wrapper. Refuses to overwrite a populated database
+# unless FORCE=1, so an accidental restore cannot silently clobber live data.
+#
+# Usage:
+#   DATABASE_URL=postgres://user:pass@host:5432/pcmi \
+#     ./scripts/backup/pcmi_restore.sh <backup.dump>
+#
+# Env:
+#   DATABASE_URL   required — target libpq connection URI
+#   FORCE          set to 1 to restore over a non-empty target (drops+recreates
+#                  objects via pg_restore --clean --if-exists)
+set -euo pipefail
+
+BACKUP_FILE="${1:-}"
+DATABASE_URL="${DATABASE_URL:-}"
+FORCE="${FORCE:-0}"
+
+if [ -z "$BACKUP_FILE" ]; then
+  echo "usage: pcmi_restore.sh <backup.dump>   (DATABASE_URL env required)" >&2
+  exit 2
+fi
+if [ ! -f "$BACKUP_FILE" ]; then
+  echo "[restore] no such file: $BACKUP_FILE" >&2
+  exit 2
+fi
+if [ -z "$DATABASE_URL" ]; then
+  echo "[restore] DATABASE_URL is required" >&2
+  exit 2
+fi
+if ! command -v pg_restore >/dev/null 2>&1 || ! command -v psql >/dev/null 2>&1; then
+  echo "[restore] pg_restore/psql not found — install postgresql-client" >&2
+  exit 2
+fi
+
+# Safety gate: count existing tables in the public schema.
+existing="$(psql "$DATABASE_URL" -tAc \
+  "SELECT count(*) FROM information_schema.tables WHERE table_schema='public'" 2>/dev/null || echo 0)"
+existing="$(echo "$existing" | tr -d '[:space:]')"
+if [ "${existing:-0}" -gt 0 ] && [ "$FORCE" != "1" ]; then
+  echo "[restore] target already has ${existing} public tables." >&2
+  echo "[restore] refusing to overwrite — re-run with FORCE=1 to proceed." >&2
+  exit 3
+fi
+
+echo "[restore] pg_restore ← $BACKUP_FILE" >&2
+# --clean --if-exists: drop existing objects first (safe on an empty DB too).
+# --no-owner/--no-privileges: restore regardless of the dump's original roles.
+# --exit-on-error keeps a partial/corrupt restore from looking successful.
+pg_restore \
+  --clean --if-exists \
+  --no-owner --no-privileges \
+  --exit-on-error \
+  --dbname="$DATABASE_URL" \
+  "$BACKUP_FILE"
+
+echo "[restore] done" >&2