Priority 4 (MCP roadmap): option_minute_paths substrate + labeler outage fix

docs/DATA-CONTRACTS.md

@@ -163,6 +163,12 @@ Every column belongs to exactly one group. The classification is written into th
 
 **Known data-quality caveats:** ~145 duplicate rows (a real finding, not a build blocker; `stg_enriched_option_outcomes` dedups to latest by `labeled_at`); ~27.7% of rows are `INVALID_LIQUIDITY` NULL-label (non-random illiquid tail — document the exclusion in any screen); pre-2026-06-11 daily counts are uneven.
 
+## Minute paths — `profitscout-fida8.profit_scout.option_minute_paths` (added 2026-07-07)
+
+Per-minute option-premium OHLCV bars over each enriched-pool candidate's **3-trading-day excursion window** (`[entry_day .. exit_day_3d]`), one row per `(contract, entry_day, ts)` with `bar_date` (ET session) and `day_index` (1–3). This is the first-crossing substrate behind the MCP's `replay_contract` and `estimate_exit_rule`'s exact-resolution + trailing-rule scoring (must-fix #6g / RM-002 / TF-14). Writers: one-shot backfill (`scripts/ledger_and_tracking/backfill_option_minute_paths.py`, executed 2026-07-07) + `forward-paper-trader/minute_paths.py` via `POST /persist_minute_paths` (daily evening cron, reconciles the last 3 scan_dates DELETE-then-LOAD). **LOAD JOBS ONLY with the explicit schema — no streaming (the reconcile DELETE would hit buffers), no autodetect.** Partition `entry_day`, cluster `contract`. Schema source of truth: `scripts/ledger_and_tracking/create_option_minute_paths.py`.
+
+**Classification: REALIZED TAPE — never a feature.** Same class as `opp_*`: research/label substrate for CLOSED windows only. Never joined into `enriched_features_v1` or any as-of ≤ scan_date surface; never read by selection or the live trader. See `docs/DECISIONS/2026-07-07-option-minute-paths.md`.
+
 ## Pool-liquidity telemetry — `profitscout-fida8.profit_scout.pool_liquidity_snapshot` (added 2026-07-07)
 
 Interval liquidity re-read of the **current enriched pool** (~50 contracts), one row per `(contract, as_of)`, written every ~10 minutes during RTH (plus one pre-open pass, `is_preopen=true`) by `signal-notifier/pool_liquidity.py` via `POST /refresh_pool_liquidity` (Cloud Scheduler `pool-liquidity-refresh`, `*/10 9-16 * * 1-5` ET). Consumed CACHE-FIRST by the gammarips-mcp `get_contract_snapshot` / `get_pool_liquidity` tools so an agent shortlist refreshes in one call at the ~10:00 ET decision window. See `docs/DECISIONS/2026-07-07-pool-liquidity-snapshot.md`.

docs/DECISIONS/2026-07-07-labeler-staging-autodetect-outage.md

@@ -0,0 +1,60 @@
+# 2026-07-07 — Labeler staging autodetect outage (07-02→07-06) and fix
+
+## Status
+Fixed 2026-07-07 (`forward-paper-trader/main.py:_write_shadow_records` step 2)
+and RECOVERED: scans 2026-07-01 (50 labeled, 8W/37L) and 2026-07-02
+(50 labeled, 22W/26L) re-run locally through the fixed path; substrate current
+again. Incremental gammarips-review: SHIP.
+
+## What happened
+The daily `/label_enriched_pool` cron failed every evening from 2026-07-02
+through 2026-07-06 with:
+
+```
+400 Provided Schema does not match Table ..._stg_enriched_option_outcomes_...
+Field recommended_spread_pct has changed type from FLOAT to STRING
+```
+
+`enriched_option_outcomes` silently stalled at scan 2026-06-30 — the paid
+substrate's labels stopped accruing for four scan days (the failure DID
+return 500s to the cron per must-fix #3a, but nobody was watching the pager).
+
+## Root cause — the 2026-07-02 landmine class, second location
+This is the same autodetect-mistypes-all-NULL-columns bug that broke
+enrichment on 2026-07-02 (`2026-07-02` decision / `3a04bee`), recurring in
+the OTHER staged-write path. The atomic shadow writer (2026-07-01) created
+its staging table correctly (`CREATE TABLE LIKE` the target → FLOAT), but the
+load used `autodetect=True` + `ALLOW_FIELD_ADDITION`. When the 07-02
+enrichment fix made `recommended_spread_pct` properly all-NULL, every staged
+JSONL load carried only `null`s for that field — BigQuery autodetect infers
+an all-null field as STRING, proposed a FLOAT→STRING relaxation against the
+typed staging table, and the load 500'd. The two fixes were siblings: the
+07-02 fix removed autodetect from enrichment's staging load; this one removes
+it from the labeler's.
+
+## Fix
+`_write_shadow_records` step 2 now:
+1. Diffs the row dicts' keys against the staging schema (which is LIKE-cloned
+   from the target, so existing columns keep live types).
+2. For genuinely NEW keys, `ALTER TABLE ADD COLUMN` on STAGING with the type
+   derived from the PYTHON values (bool→BOOL before int→INT64 — bool is an
+   int subclass — float→FLOAT64, else STRING). Deterministic typing from code
+   we control, not BQ inference over serialized JSON.
+3. Loads with `schema=<staging schema>` explicitly. No autodetect, no
+   ALLOW_FIELD_ADDITION.
+
+All atomicity guarantees unchanged (staging-only until the verified
+transactional swap; failure = loud stall, target untouched). Applies to all
+three `_write_shadow_records` callers (enriched outcomes, topscore shadow,
+shadow intraday).
+
+## Rule (now enforced at both known locations)
+**NEVER autodetect on a staged load, anywhere.** An all-NULL column — which
+`recommended_spread_pct` permanently is on this data plan — autodetects to
+STRING and poisons the load. If a third staged-write path is ever added, it
+inherits this rule.
+
+## Watch item
+The evening cron will label scan 2026-07-06 tonight (17:00 ET) — it needs the
+FIXED code deployed on forward-paper-trader before then, or one more manual
+re-run tomorrow.

docs/DECISIONS/2026-07-07-option-minute-paths.md

@@ -0,0 +1,86 @@
+# 2026-07-07 — `option_minute_paths` companion table (must-fix #6g / MCP RM-002 + TF-14)
+
+## Status
+Shipped 2026-07-07. Table created + FULL history backfilled (144,448 bars,
+2,675 contract-days, entry days 2026-04-13 → 2026-07-01; 419 contract-days
+legitimately empty — the no-print illiquid tail) + topped up through
+2026-07-06. Daily top-up endpoint on forward-paper-trader
+(`POST /persist_minute_paths`) + Cloud Scheduler `option-minute-paths-refresh`.
+
+## Context
+The excursion substrate stored only MFE/MAE **extremes** (`opp_*`), so when a
+bracket's target AND stop were both crossed inside the window, first-crossing
+order was unrecoverable — `estimate_exit_rule` resolved those rows by a
+minutes-to-extreme heuristic (~10.5% of classifications), and trailing rules
+could not be scored at all. This was engine must-fix #6g (known-deferred) and
+the trader harness's GAP-002.
+
+## Decision
+Persist the minute tape the labeler already replays from:
+
+1. **`profit_scout.option_minute_paths`** — one row per (contract, entry_day,
+   ts): per-minute OHLCV premium bars over each enriched-pool candidate's
+   3-trading-day excursion window. Partition `entry_day`, cluster `contract`.
+   DDL: `scripts/ledger_and_tracking/create_option_minute_paths.py`.
+   **LOAD JOBS ONLY with the explicit schema** (daily reconcile DELETEs by
+   scan_date — streaming buffers would block that; autodetect banned per the
+   2026-07-02 outage rule).
+2. **Backfill** — `scripts/ledger_and_tracking/backfill_option_minute_paths.py`
+   (executed 2026-07-07): one Polygon minute-aggs range call per labeled
+   (contract, entry_day) from `enriched_option_outcomes`.
+3. **Daily top-up** — `forward-paper-trader/minute_paths.py` behind
+   `POST /persist_minute_paths`: each evening reconciles the last 3
+   scan_dates' pools (window `[entry_day .. min(exit_day_3d, today)]`,
+   DELETE-then-LOAD per scan_date, idempotent). Pool comes from
+   `overnight_signals_enriched` so it is independent of labeler lag. Fully
+   walled: called only from its own endpoint; the trader/labeler paths never
+   import it. Token-gated with the same `POOL_LIQ_REFRESH_TOKEN` secret as
+   the notifier's pool-liquidity endpoint (scan_dates override refused
+   without the token).
+4. **Cloud Scheduler `option-minute-paths-refresh`** — daily ~17:40 ET
+   weekdays (after the 17:00 labeler + 17:20 outcomes refresh), with the
+   `X-Refresh-Token` header.
+
+## What it unlocks (MCP)
+- **`replay_contract(contract, date, target_pct?, stop_pct?)`** — per-session
+  minute path served table-first (upstream fallback for non-pool contracts),
+  with an optional exact first-crossing readout off the 10:00 ET anchor.
+- **`estimate_exit_rule` exact resolution** — both-levels-crossed rows now
+  resolve by exact first touch (same-bar → STOP-first, the labeler's
+  pessimistic rule). Measured on the +40/−30 3d bracket: heuristic share
+  0.1045 → **0.0** (212/212 ambiguous rows resolved; 90 TARGET / 122 STOP).
+- **`estimate_exit_rule(rule='trailing')` (TF-14)** — bar-by-bar SQL replay
+  of a hard-stop + trailing-giveback (+optional activation) rule, horizons
+  same_day (day-1 bars ≤15:45 ET) and 3d. Conservative bar mechanics: trail
+  rides the PRIOR-bar peak; touch fills at min(bar open, level). The entry
+  anchor (`opp_entry_price`) already embeds the engine's entry-slippage fill;
+  no EXIT slippage is applied (results are entry-net, exit-gross).
+
+## Leakage classification
+**REALIZED TAPE — never a feature.** Same class as the `opp_*` opportunity
+surface: research/label substrate for CLOSED windows. Never joined into
+`enriched_features_v1` or any as-of ≤ scan_date surface; never read by the
+selection path or the live trader. Serving closed-window bars to agents is
+data-vendor product (the historical tape, explicitly timestamped), consistent
+with the opportunity-surface posture.
+
+## Recovery path
+If a reconcile dies between its DELETE and LOAD on a window's LAST (day-3)
+pass, that scan_date drops out of the next evening's last-3 set and stays
+empty until manually re-run. Recovery: POST /persist_minute_paths with the
+`X-Refresh-Token` header and `{"scan_dates": ["YYYY-MM-DD"]}` (token-gated
+override, ≤10 dates). `--max-instances=1` + the daily cadence make the
+overlap/double-run case remote.
+
+## Cost / blast radius
+Backfill ≈ 3.1k Polygon calls (one-time). Daily ≈ 150 calls + ~6k rows.
+Failure mode = stale minute paths (MCP replay falls back upstream; exit-rule
+scoring reports coverage via `n_excluded_no_bars` / heuristic share) — never
+a missed pick, never a broken label run. Kill switch = pause the Scheduler job.
+
+## Revisit when
+- The pool cap or window length changes (row volume scales linearly).
+- A real quote feed lands (bars could then carry NBBO context — new columns,
+  explicit schema change, never autodetect).
+- Same-day arbitrary-bracket scoring is requested (the minute data now
+  supports it; deliberately not built in this pass).

forward-paper-trader/deploy.sh

@@ -8,6 +8,12 @@ SERVICE_NAME="forward-paper-trader"
 
 echo "Deploying $SERVICE_NAME to Cloud Run in project $PROJECT_ID..."
 
+# POOL_LIQ_REFRESH_TOKEN (2026-07-07): shared refresh token for the
+# POST /persist_minute_paths research endpoint (X-Refresh-Token header; same
+# secret the notifier's /refresh_pool_liquidity uses). MUST stay in
+# --set-secrets — the flag REPLACES the secret set, so dropping it silently
+# strips the mount on the next deploy.
+
 gcloud run deploy $SERVICE_NAME \
   --project=$PROJECT_ID \
   --region=$REGION \
@@ -20,7 +26,7 @@ gcloud run deploy $SERVICE_NAME \
   --min-instances=0 \
   --max-instances=1 \
   --set-env-vars="GCP_PROJECT_ID=$PROJECT_ID" \
-  --set-secrets="POLYGON_API_KEY=POLYGON_API_KEY:latest" \
+  --set-secrets="POLYGON_API_KEY=POLYGON_API_KEY:latest,POOL_LIQ_REFRESH_TOKEN=POOL_LIQ_REFRESH_TOKEN:latest" \
   --service-account="firebase-adminsdk-fbsvc@$PROJECT_ID.iam.gserviceaccount.com"
 
 echo "Done!"

forward-paper-trader/main.py

@@ -1584,10 +1584,15 @@ def _write_shadow_records(
     committed — silently wiping that scan_date's rows with nothing to reload. New
     rows are now STAGED first, verified, then the target scan_date is replaced
     inside a single transaction that rolls back on any error, so the original rows
-    always survive a failure. autodetect=True keeps a newly-added feature column
-    from 500-ing the load; any new column is propagated onto the target BEFORE the
-    swap. When no new columns are present this is byte-for-byte equivalent to the
-    old delete-then-overwrite (scan_date fully replaced, no dups).
+    always survive a failure. Schema drift is handled WITHOUT autodetect
+    (2026-07-07 — autodetect inferred the all-NULL recommended_spread_pct as
+    STRING and 500'd every load, labeler down 07-02..07-06): new row keys are
+    diffed against the staging schema and ADDed as columns typed from the
+    Python values, then the load binds the staging table's explicit schema;
+    any new column is propagated onto the target BEFORE the swap. When no new
+    columns are present this is byte-for-byte equivalent to the old
+    delete-then-overwrite (scan_date fully replaced, no dups). NEVER re-add
+    autodetect here — see docs/DECISIONS/2026-07-07-labeler-staging-autodetect-outage.md.
 
     Mirrors _write_ledger_records' streaming-avoiding load-job pattern but takes
     an explicit ``table`` — deliberately NOT reusing _write_ledger_records (which
@@ -1631,17 +1636,54 @@ def _write_shadow_records(
     ).result()
 
     try:
-        # 2) Load new rows into staging. autodetect=True + ALLOW_FIELD_ADDITION
-        #    means a genuinely NEW field is ADDED to staging instead of 500-ing.
+        # 2) Load new rows into staging with the staging table's EXPLICIT
+        #    schema — NEVER autodetect (2026-07-07 fix): with autodetect=True,
+        #    an all-NULL column (recommended_spread_pct went permanently NULL
+        #    with the 2026-07-02 enrichment fix) is inferred as STRING, which
+        #    conflicts with the staging table's FLOAT and 500s EVERY load —
+        #    the labeler was down 07-02..07-06 on exactly this (the same
+        #    landmine class as the 07-02 enrichment outage, second location).
+        #    Schema-drift tolerance (the reason autodetect was added 07-01) is
+        #    preserved by diffing row keys against the staging schema and
+        #    ADDing columns typed from the PYTHON values before the load —
+        #    deterministic typing from code we control, not BQ inference over
+        #    serialized JSON.
+        staging_schema = client.get_table(staging).schema
+        known_names = {f.name for f in staging_schema}
+        new_keys = sorted({k for r in rows for k in r} - known_names)
+        if new_keys:
+
+            def _bq_type_of(key: str) -> str:
+                for r in rows:
+                    v = r.get(key)
+                    if v is None:
+                        continue
+                    if isinstance(v, bool):
+                        return "BOOL"
+                    if isinstance(v, int):
+                        return "INT64"
+                    if isinstance(v, float):
+                        return "FLOAT64"
+                    return "STRING"  # str/date/datetime/etc — safe default
+                return "STRING"
+
+            adds = ", ".join(
+                f"ADD COLUMN IF NOT EXISTS `{k}` {_bq_type_of(k)}" for k in new_keys
+            )
+            client.query(f"ALTER TABLE `{staging}` {adds}").result()
+            staging_schema = client.get_table(staging).schema
+            logger.info(
+                f"shadow: staged {len(new_keys)} new drift column(s): {new_keys}"
+            )
+
         jsonl = "\n".join(json.dumps(r, default=str) for r in rows)
         load_job = client.load_table_from_file(
             io.BytesIO(jsonl.encode("utf-8")),
             staging,
             job_config=bigquery.LoadJobConfig(
                 write_disposition=bigquery.WriteDisposition.WRITE_APPEND,
                 source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,
-                schema_update_options=[bigquery.SchemaUpdateOption.ALLOW_FIELD_ADDITION],
-                autodetect=True,
+                schema=staging_schema,
             ),
         )
         load_job.result()  # raises on load failure -> target still untouched
@@ -2640,6 +2682,82 @@ def trigger_paper_trading():
         return jsonify({"status": "error", "message": str(e)}), 500
 
 
+@app.route("/persist_minute_paths", methods=["POST"])
+def trigger_persist_minute_paths():
+    """RESEARCH-ONLY minute-path top-up (MCP Priority-4 / must-fix #6g).
+
+    Reconciles `option_minute_paths` for the last 3 scan_dates' enriched
+    pools (each window [entry_day .. min(exit_day_3d, today)]); by a window's
+    third session it is complete and drops out of the set. Hit by the daily
+    `option-minute-paths-refresh` Cloud Scheduler cron in the evening, after
+    the labeler. Fully independent of the trader/labeler paths — fetch +
+    write logic lives in minute_paths.py behind a hard leakage wall (realized
+    tape; never a feature; never read by selection or the live trader).
+
+    Token-gated like the notifier's pool-liquidity endpoint: when
+    POOL_LIQ_REFRESH_TOKEN is set, every call must send X-Refresh-Token, and
+    the scan_dates override is refused without it (fail-closed while services
+    are public; see docs/DECISIONS/2026-07-02-service-auth-hardening.md).
+    Body (optional, token-gated): {"scan_dates": ["YYYY-MM-DD", ...]}.
+    """
+    import hmac as _hmac
+
+    import minute_paths
+
+    expected_token = os.environ.get("POOL_LIQ_REFRESH_TOKEN", "").strip()
+    provided_token = request.headers.get("X-Refresh-Token", "")
+    token_authed = bool(expected_token) and _hmac.compare_digest(
+        provided_token, expected_token
+    )
+    if expected_token and not token_authed:
+        return jsonify({"status": "denied"}), 403
+
+    req = request.get_json(silent=True) or {}
+    if req.get("scan_dates") and not token_authed:
+        return jsonify(
+            {"status": "denied", "reason": "scan_dates override requires X-Refresh-Token"}
+        ), 403
+
+    try:
+        today_et = datetime.now(est).date()
+        if req.get("scan_dates"):
+            try:
+                scan_dates = [
+                    datetime.strptime(str(sd), "%Y-%m-%d").date()
+                    for sd in req["scan_dates"]
+                ][:10]
+            except ValueError:
+                return jsonify({"status": "error", "reason": "scan_dates must be YYYY-MM-DD"}), 400
+        else:
+            # the last 3 trading days strictly before today — the scan_dates
+            # whose 3-session excursion windows are still accreting bars
+            scan_dates = []
+            d = today_et - timedelta(days=1)
+            while len(scan_dates) < 3 and d > today_et - timedelta(days=15):
+                if is_trading_day(d):
+                    scan_dates.append(d)
+                d -= timedelta(days=1)
+
+        windows = []
+        for sd in scan_dates:
+            entry_day = get_next_trading_day(sd)
+            if entry_day > today_et:
+                continue
+            window_end = min(get_nth_next_trading_day(entry_day, 2), today_et)
+            windows.append(
+                {"scan_date": sd, "entry_day": entry_day, "window_end": window_end}
+            )
+        if not windows:
+            return jsonify({"status": "skipped", "reason": "no closed windows to persist"}), 200
+
+        summary = minute_paths.persist_minute_paths(windows)
+        code = 200 if summary.get("status") in ("success", "partial") else 500
+        return jsonify(summary), code
+    except Exception as e:
+        logger.error(f"Error in persist_minute_paths endpoint: {e}")
+        return jsonify({"status": "error", "message": str(e)}), 500
+
+
 @app.route("/label_enriched_pool", methods=["POST", "GET"])
 def trigger_label_enriched_pool():
     """RESEARCH-ONLY counterfactual labeling endpoint.