Skip to content

Commit 4ce77cb

Browse files
anuveyatsuclaude
andcommitted
feat(skills): arcgis-to-portaljs Phase 3 — sync mode + cutover design (po-0qe.3)
Build out Phase 3 of the ArcGIS Hub migrator from a placeholder table row into a full, reviewable design + copy-paste templates. Live execution (re-migration, cron wiring) is split into child beads po-0qe.3.1 / .3.2. - NEW references/sync-and-cutover.md: (a) scheduled sync — CI-cron-not-Worker rationale, arcgis-sync-state.json ledger, per-layer drift detection via editingInfo.dataLastEditDate OR returnCountOnly delta, idempotent upsert, GH Actions workflow template, miss-count safety; (b) parity dashboard over an append-only history ledger; (c) cutover playbook — slug preservation, 301s when you own the domain, sitemap + crosswalk + canonical + link-out when you don't (Lewisville opendata.arcgis.com); (d) post-cutover options matrix. - commands/arcgis-to-portaljs.md: add operator-facing §10; §7 now emits the sync ledger so the shadow is re-syncable at no extra cost. - reference.md phasing table (Phase 3 -> Yes) + SKILL.md pointer. Design grounded in the live Lewisville Hub: editingInfo.dataLastEditDate === DCAT modified, and Parcels drift (30,694 -> 30,696) is caught by both signals. gen:skills:check passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 5d3d94c commit 4ce77cb

4 files changed

Lines changed: 337 additions & 3 deletions

File tree

.claude/commands/arcgis-to-portaljs.md

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -401,6 +401,12 @@ renders it as a separate "Migrated" field so it never masquerades as the data's
401401
Set `migratedAt` once per run from a fixed `date -u +%FT%TZ` captured at the start, so re-runs
402402
are stable.
403403

404+
**Also write the sync ledger `arcgis-sync-state.json`** — one entry per migrated layer keyed on
405+
`(namespace, slug)` carrying `serviceUrl`, `recordCount` (= `TOTAL`), `dataLastEditDate` (from the
406+
layer `?f=json` `editingInfo`), `extent`, and `migratedAt`. Phase 3 sync (§10) reads it to detect
407+
per-layer drift cheaply on a schedule; producing it here is what makes the shadow re-syncable at no
408+
extra cost. See [`sync-and-cutover.md`](https://github.com/datopian/portaljs/blob/main/skills/arcgis-to-portaljs/references/sync-and-cutover.md).
409+
404410
`--dry-run` stops before any write/push and prints the plan (mirror `/portaljs-migrate` §6).
405411

406412
### 8. Parity report (source vs derived) — the migration promise
@@ -457,9 +463,54 @@ even when a few layers fail — never suppress a mismatch.
457463
- Parity: <P> PASS / <W> WARN / <F> failed → arcgis-parity-report.md
458464
- Showcase: /@<namespace>/<slug> — <MapPreview> + <GeoQuery> per vector dataset
459465
Next: review arcgis-parity-report.md, then `npm run dev` to click through the top datasets.
460-
Deploy with /portaljs-deploy. (Sync + cutover = Phase 3; see the child beads on po-0qe.)
466+
Deploy with /portaljs-deploy. Keep it current + retire the old site with §10 (Phase 3).
467+
```
468+
469+
### 10. Sync mode & cutover (Phase 3) — keep the shadow current, retire the old site
470+
471+
Phase 0 migrates **once**. Phase 3 keeps the PortalJS shadow tracking the live Hub and preserves old
472+
URLs at cutover. Full design + copy-paste templates:
473+
[`sync-and-cutover.md`](https://github.com/datopian/portaljs/blob/main/skills/arcgis-to-portaljs/references/sync-and-cutover.md).
474+
Mental model is unchanged — **AGOL edits, PortalJS shadows**.
475+
476+
**(a) Scheduled sync.** Write a sync ledger `arcgis-sync-state.json` in §7 (one entry per layer:
477+
`serviceUrl`, `recordCount`, `dataLastEditDate`, `extent`, `migratedAt`). A cron re-runs §5→§7 **only
478+
for layers that drifted**. Detect drift cheaply per layer before any export:
479+
480+
```bash
481+
LIVE_EDIT=$(curl -sS -m30 "$SVC?f=json" | jq -r '.editingInfo.dataLastEditDate // .editingInfo.lastEditDate // empty')
482+
LIVE_COUNT=$(curl -sS -m30 "$SVC/query?where=1=1&returnCountOnly=true&f=json" | jq -r '.count')
483+
# changed if LIVE_EDIT > stored edit OR LIVE_COUNT != stored count
461484
```
462485

486+
Keep **both** signals: `editingInfo` is authoritative but null when a service disables editor
487+
tracking; the count delta is the always-available fallback (and catches silent add/delete). Upsert on
488+
`(namespace, slug)` — unchanged layers do no work. Run it in **CI cron (GitHub Actions), not a
489+
Worker** — the conversion tier is native `ogr2ogr`/`tippecanoe`/`duckdb`, which a Worker can't run;
490+
the workflow installs the toolchain, runs the detect-and-apply loop, commits `datasets.json` +
491+
`data/**` back, redeploys. Never delete a layer on a single miss (track `missCount`, flag at ≥3).
492+
493+
**(b) Parity dashboard.** Append one row per layer per run to `arcgis-parity-history.jsonl`
494+
(`{run, slug, source, derived, verdict, edit}`); a small node script folds it into
495+
`arcgis-parity-dashboard.md` (counts, drift-since-first-migration, last sync, verdict streak).
496+
Optionally surface it as a `/sync-status` content page so "how current is this portal?" is answerable
497+
from the site.
498+
499+
**(c) Cutover — preserve `…/datasets/<slug>` links.** Preserve slugs (map the Hub slug → showcase
500+
slug at migration; don't re-derive). Then:
501+
- **You own the domain** (custom Hub domain / Stream) → emit 301s old→new (Cloudflare Bulk Redirects
502+
or Next `redirects()`), generated from the DCAT `landingPage` × `datasets.json` crosswalk.
503+
- **You don't** (Lewisville is on `opendata.arcgis.com` — Esri controls it, no redirects possible) →
504+
publish `sitemap.xml` (submit to Search Console), a public **crosswalk page** (old item → new URL),
505+
self-`canonical` tags, and request the city add a "moved to <url>" link-out on the Hub items. Track
506+
it in `cutover-checklist.md`.
507+
508+
**(d) Post-cutover options.** Document three futures for the owner: **1)** keep AGOL upstream (status
509+
quo — PortalJS is a free public read-shadow, still paying Esri); **2)** git-native publishing per
510+
layer (QGIS/CSV → git, portal becomes source of truth, low-churn layers first); **3)** full Esri exit
511+
(open stack; a Koop FeatureServer facade — Phase 4 — keeps old REST consumers working during the
512+
window). Recommend **1 → 2 (layer by layer) → 3**, never big-bang.
513+
463514
## Error handling
464515

465516
```

skills/arcgis-to-portaljs/SKILL.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,5 +111,6 @@ The canonical, full step-by-step workflow is
111111

112112
- Full workflow: [`.claude/commands/arcgis-to-portaljs.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/arcgis-to-portaljs.md)
113113
- REST export, classification, parity, and phasing details: [`references/reference.md`](references/reference.md)
114+
- Ongoing sync, parity dashboard, and cutover (Phase 3): [`references/sync-and-cutover.md`](references/sync-and-cutover.md)
114115
- Related skills: `portaljs-migrate`, `portaljs-add-geo`, `portaljs-add-dataset`, `portaljs-deploy`
115116
- ArcGIS REST query API: <https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer/> · tippecanoe: <https://github.com/felt/tippecanoe> · DuckDB spatial: <https://duckdb.org/docs/extensions/spatial>

skills/arcgis-to-portaljs/references/reference.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,9 @@ one push (bytes → Giftless/R2, no GitHub remote required) → `git lfs prune`.
7979
`${R2_PUBLIC_BASE:-https://data.portaljs.com}/lfs/datopian/<slug>/<oid>`. OSS self-host signs
8080
locally with `giftless/mint-token.py`. Dual-tier `datasets.json` entry shape: see
8181
`portaljs-add-geo` §10 (`tiles` pmtiles + `geo` geoparquet + `source` original, with a
82-
bbox-first `query` filled from the layer extent).
82+
bbox-first `query` filled from the layer extent). Also emit the sync ledger
83+
`arcgis-sync-state.json` (per-layer `serviceUrl`/`recordCount`/`dataLastEditDate`/`extent`) so the
84+
Phase-3 scheduled sync can detect drift — see [`sync-and-cutover.md`](sync-and-cutover.md).
8385

8486
## Phasing (epic po-0qe §6)
8587

@@ -88,7 +90,7 @@ bbox-first `query` filled from the layer extent).
8890
| 0 | Harvest + export + convert + publish + parity (the automatable core) | **Yes** |
8991
| 1 | Lewisville shadow portal (single-publisher, geo-heavy) | Run the skill; own bead |
9092
| 2 | Stream (multi-publisher orgs, content pages, triage workflow) | Own bead |
91-
| 3 | Scheduled sync + cutover playbook (re-harvest, redirects, DNS) | Own bead |
93+
| 3 | Scheduled sync + cutover playbook (re-harvest, redirects, DNS) | **Yes**[`references/sync-and-cutover.md`](sync-and-cutover.md) |
9294
| 4 | Koop FeatureServer-compat adapter (keep old REST consumers) | Own bead (optional) |
9395

9496
3D scenes / StoryMaps / drone imagery have no serverless open equivalent — link-out or leave

0 commit comments

Comments
 (0)