Devshard Gateway Postgres Backend

[a-kuprin]

Summary

devshardctl gateway management state (settings, devshards, throttles, rotation
status, suspicious hosts, escrow commitments) can now persist to PostgreSQL
when PGHOST is set, with SQLite as a durable fallback. The design mirrors
the hybrid pattern already used for payload storage: Postgres is primary during
normal operation; SQLite absorbs writes when Postgres is unavailable and is used
for reads when Postgres is empty or errors.

Activation is env-driven (PGHOST set → hybrid; unset → SQLite-only). A
one-shot startup migration copies an existing gateway.db into Postgres on first
connect. A sync journal (SQLite outbox) records only the rows changed during
a Postgres outage and replays them to Postgres on reconnect — before Postgres
resumes as primary — so we never blindly overwrite healthy PG data from stale
SQLite.

Also makes PG_CONNECT_TIMEOUT configurable for decentralized-api payload
storage (unchanged behavior when unset).

Architecture

                    ┌─────────────────┐
  write request ──► │ HybridGatewayStore │
                    └────────┬────────┘
                             │
              PG healthy?    │
                    ┌────────┴────────┐
                    ▼                 ▼
            PostgresGatewayStore   SQLiteGatewayStore
            (primary)              (+ sync journal on fallback)

Healthy path. Writes and reads go to Postgres. SQLite is not updated on every
write — it is a fallback copy, not a live replica.

Fallback path. On Postgres failure (after in-request retries), the write goes
to SQLite and a journal entry (table, row_key, op) is appended in the same
SQLite transaction as the data change.

Recovery path. When Postgres comes back, the hybrid store drains the journal
(coalesce last op per key → apply upserts/deletes to Postgres in one PG
transaction → delete drained journal rows) and only then publishes the Postgres
connection as primary again.

What was added

Piece	Role
PostgresGatewayStore	PG implementation of GatewayStore
SQLiteGatewayStore	Extracted SQLite implementation (was inline)
HybridGatewayStore	Routes reads/writes; lazy reconnect; journal integration
MigrateGatewaySQLiteToPostgres	One-shot idempotent import of existing gateway.db
gateway_pg_sync_journal	SQLite outbox for outage-era deltas
NewGatewayStore factory	Selects backend from env; runs migration + initial drain

Edge cases and how they are handled

Brief Postgres blip (sub-second reset / failover)

A single failed write does not immediately fall back. PG_WRITE_RETRY_BUDGET
(default 2s) retries with exponential backoff so pgxpool can hand out a
fresh connection. Each attempt is bounded by PG_OPERATION_TIMEOUT (default
2s) so a hung query cannot block indefinitely.

If all retries succeed, the write never touches SQLite or the journal.

Sustained Postgres outage

After the retry budget is exhausted, the hybrid store drops the cached Postgres
connection (dropPg) and routes writes to SQLite + journal. This is required:
if we kept h.pg set, pgxpool could transparently reconnect and serve PG writes
while stale journal entries still existed — a later drain would replay outdated
SQLite rows over newer PG data (lost update).

While h.pg is nil, writes skip Postgres entirely (fast SQLite path). Reconnect
is rate-limited by PG_RETRY_INTERVAL (gateway default 15s).

PG commit succeeds but journal delete fails

Drain order: apply to Postgres → commit PG tx → delete journal rows up to
max(seq). If the delete fails or the process crashes in between, entries remain
and are re-applied on the next drain. Replays are idempotent (ON CONFLICT DO UPDATE; deletes are no-ops if absent). Journal seq is AUTOINCREMENT, so
post-snapshot writes always get a higher seq and are not accidentally deleted.

Journal write vs data write atomicity

writeWithSyncJournal runs the data mutation and journal inserts in one SQLite
transaction. A failure rolls back both — no orphan journal rows, no journaled
changes without persisted data.

Reads needed inside that transaction (e.g. returning the updated suspicious-host
list) use the transaction handle, not a separate db.Query, because SQLite is
capped at SetMaxOpenConns(1) and a second connection would deadlock.

Split-brain / stale SQLite after healthy PG period

During healthy operation SQLite is not kept in sync. A naive "copy all of
SQLite back to Postgres on reconnect" would clobber PG-only updates from before
the outage. The journal records only keys mutated while Postgres was down;
drain replays those keys' current SQLite row (last-writer-wins per key), not
a full snapshot.

One-shot migration vs journal

Migration (MigrateGatewaySQLiteToPostgres) is the bulk initial seed when
enabling Postgres on an existing gateway.db (guarded by marker + PG emptiness).
The journal handles incremental outage deltas afterward. Both use the same
idempotent per-row upsert helpers; overlap is harmless.

Process restart during outage

The journal lives in SQLite and survives restarts. On startup with PGHOST set,
the factory connects to Postgres, runs migration if needed, then drains any
pending journal before serving traffic.

Rollback to SQLite-only (PGHOST unset after PG period)

If an operator disables Postgres after running with it, PG-only writes are not
in SQLite — this is an explicit escape hatch, not automatic bidirectional
sync. Re-enabling Postgres later relies on PG as source of truth plus any journal
entries from subsequent outages. Documented as an operator caveat.

Drain blocks concurrent publish

Journal drain runs under the hybrid mutex; h.pg is set only after the journal is
empty. Concurrent writers also take the mutex when probing reconnect, so no new
journal entry can slip in between "journal empty" and "PG published". Steady-state
hot path only holds the mutex to read the connection pointer — not across I/O.

Kill switch: Postgres-only mode

PG_TO_SQLITE_FALLBACK=false disables SQLite fallback on Postgres errors. Writes
and reads return errors when Postgres is unavailable; startup fails if Postgres
cannot be reached. The sync journal is not used in this mode. Use when Postgres
is the sole source of truth and operators prefer hard failures over degraded
SQLite service.