Retire mistaken Hetzner OS backups (#6068)

Author: viktor-shcherb

apps/crawler/VERSION

@@ -1 +1 @@
-0.13.189
+0.13.190

apps/crawler/alerts.yaml

@@ -194,7 +194,7 @@ groups:
           route: codex-daily
         annotations:
           summary: "{{ $labels.service }} backup failed on {{ $labels.instance }}"
-          description: "The latest application-data backup attempt failed. Legacy OS backups must remain enabled."
+          description: "The latest application-data backup attempt failed. The native repository is now the only recovery path; investigate immediately and preserve existing artifacts."
           runbook: https://github.com/colophon-group/jobseek/blob/main/docs/19-data-backup-recovery.md#failure-and-removal-gates
 
       - alert: DataBackupStale

docs/19-data-backup-recovery.md

@@ -71,9 +71,9 @@ The root-owned fleet sampler republishes only the numeric status fields as
 `jobseek_backup_*` metrics with stable host/service labels; it never forwards
 the JSON error text. `DataBackupFailed` and `DataBackupStale` are owned by the
 daily Codex route in `apps/crawler/alerts.yaml`. This establishes the telemetry
-source, but it does not by itself satisfy the server-backup removal gate: the
-error-review bundle must still prove bounded historical reads and GitHub issue
-delivery without a write credential.
+source. Bounded historical reads and GitHub issue delivery without a write
+credential remain tracked by #5948; the owner-approved legacy-backup retirement
+described below did not waive that follow-up.
 
 ## Initial production evidence (2026-07-22)
 
@@ -94,8 +94,18 @@ failures and Typesense remained healthy without a restart.
 Both timers are enabled and their next jittered runs are visible. Delete and
 rebuild protection is enabled on both data servers, delete protection is
 enabled on the PostgreSQL Volume, and the validated pre-cutover PostgreSQL
-container has been removed. Legacy server backups remain until daily
-error-review alert delivery passes the removal gate below.
+container has been removed.
+
+On 2026-07-23, the account owner explicitly directed retirement of the mistaken
+server backups while #5948 was paused on account-owner acceptance of updated
+Grafana terms. Immediately before the control-plane change, both native backup
+jobs had succeeded that day, both timers were enabled, PostgreSQL WAL archival
+had zero failures, the Storage Box and data resources were delete-protected,
+both services were healthy with zero restarts/OOMs, and no backup alert was
+firing. Disabling the two Hetzner backup schedules also removed all seven
+server-bound backup images for each host. PostgreSQL and Typesense were not
+restarted. The independent encrypted repositories and Storage Box snapshots
+remain the recovery artifacts.
 
 ## Installation and scheduling
 
@@ -302,8 +312,7 @@ Cloudflare tunnel to a restore drill.
 
 ## Failure and removal gates
 
-Do not remove the existing Hetzner server backups until all of the following
-are true for both services:
+The normal replacement gate for any future legacy backup retirement is:
 
 - the off-host backup completed and repository validation passed;
 - an isolated restore using that repository passed;
@@ -312,12 +321,18 @@ are true for both services:
   evidence and can create or update an actionable GitHub issue;
 - recovery evidence and measured recovery time are recorded in the audit.
 
-Current gate state as of 2026-07-22: off-host backups, repository validation,
-isolated restores, measured recovery evidence, enabled schedules, and visible
-next runs have passed. Daily Codex failure/freshness evidence and proven GitHub
-issue delivery remain pending. Therefore the legacy server backups must remain
-enabled even though the replacement data protection is operating normally.
-
-After the gate passes, disable server backups and delete the residual server
-backup images for PostgreSQL and Typesense. Preserve the independent Storage
-Box repositories and their secondary snapshots.
+Current state as of 2026-07-23: off-host backups, repository validation,
+isolated restores, measured recovery evidence, enabled schedules, visible next
+runs, resource protection, and live Grafana backup series have passed. At the
+account owner's explicit direction, the two mistaken Hetzner backup schedules
+were disabled before #5948's daily issue-delivery proof; the provider action
+also removed all residual server-backup images. This is a recorded,
+owner-approved exception, not a relaxation of the normal replacement gate.
+
+Until #5948 completes, an operator must inspect the atomic backup status,
+repository validation, timers, WAL archive state, and `DataBackupFailed` /
+`DataBackupStale` rule state during production maintenance and backup incident
+review. A failure or stale result is critical because the native encrypted
+repositories are now the only recovery artifacts. Do not re-enable OS backups
+as a substitute for repairing the native data-backup path. Preserve the
+independent Storage Box repositories and their secondary snapshots.

docs/audits/2026-07-22-hetzner-systems.md

@@ -67,6 +67,26 @@ pass.
   Codex error-review workflow and can create or update an actionable GitHub
   issue, as required by `docs/19-data-backup-recovery.md`.
 
+### 2026-07-23 — mistaken OS backups retired
+
+- Immediately before the provider change, the scheduled PostgreSQL
+  differential backup and Typesense snapshot/upload/check had both succeeded
+  that day. Their timers were enabled, the next runs were visible, PostgreSQL
+  WAL archive failures were zero, the Storage Box and data resources retained
+  delete protection, both services were healthy with zero restart/OOM counts,
+  and the live backup-success/freshness series had no firing backup alert.
+- At the account owner's explicit direction, the PostgreSQL and Typesense
+  Hetzner server-backup schedules were disabled while #5948 remained paused on
+  account-owner acceptance of updated Grafana terms. Hetzner's disable action
+  also removed all seven residual server-bound images for each server. This was
+  a control-plane-only change; PostgreSQL and Typesense were not restarted.
+- The encrypted pgBackRest/WAL and Typesense Restic repositories, their tested
+  restore procedures, and the Storage Box's seven daily secondary snapshots
+  are now the recovery paths. Daily Codex issue-delivery proof remains a
+  separate high-priority operational gap under #5948; until it is complete,
+  backup state must be checked directly during production maintenance and
+  backup incident review.
+
 ### 2026-07-22 — fleet observability staged
 
 - Merged and deployed the repo-owned host telemetry surface from `main` to the
@@ -461,7 +481,8 @@ Observed project inventory for the in-scope systems: three running servers, one
 
 - Name and verify accountable human owners/on-call escalation paths; the Cloud project API does not expose project membership or billing ownership.
 - Prove database/search backup failure and freshness evidence in the daily
-  Codex issue-delivery path, then remove the mistaken Hetzner OS backups.
+  Codex issue-delivery path. The mistaken Hetzner OS backups have already been
+  retired under the recorded owner-approved exception.
 - Add explicit historical metrics coverage to the root-produced Codex error-review evidence boundary without exposing production or write credentials.
 - Verify the documented Cloudflare per-IP rate-limit rule and notification routing in the Cloudflare control plane.
 - Verify notification delivery rather than rule evaluation only; the false `ExporterStale` alerts show that firing state alone is not useful evidence.
@@ -472,4 +493,5 @@ The baseline audit phase was read-only. Subsequent operator-approved changes
 are recorded only in the verified remediation change log above so the original
 observations remain reproducible. Replacement database/search backups,
 restores, scheduling, capacity expansion, and resource protections are now
-verified; legacy OS-backup removal remains gated on daily alert-delivery proof.
+verified, and the mistaken OS backups have been retired. Daily alert-to-issue
+delivery remains pending under #5948.