Replace Phenom api_sniffer with dedicated sitemap monitor (#2538)

* Replace Phenom api_sniffer with dedicated sitemap monitor

All 9 Phenom boards were silently truncating to ~500 jobs per cycle:
the api_sniffer path hit a 50-page hard cap with Phenom's fixed
10-per-page API, and find_total_count() didn't recognise Phenom's
totalJob field so the cap-lifting logic at api_sniffer.py:1085 never
kicked in.

Two fixes, independently shippable:

1. Extend COUNT_FIELDS regex to match total<Suffix> patterns so the
   existing cap-lifting logic picks up vendor-specific names like
   Phenom's totalJob. This alone unblocks every api_sniffer board
   whose ATS exposes a non-standard total field.

2. New phenom monitor (core/monitors/phenom.py) that discovers URLs
   via the tenant's /sitemap.xml sitemap-index. Replaces the 20-min
   (Marriott) to 90-min (mchire) paginated browser fetch with a ~5s
   plain-httpx sitemap traversal. Per-URL content comes from the
   detail page's JSON-LD JobPosting via the json-ld scraper (with
   render:true for tenants whose detail pages need Playwright render).

   Sitemap-index classification is language-aware: tenants with per-
   locale children (Marriott = 22 locales, Nike = 16 locales) get
   non-English children filtered out, matching the old api_sniffer's
   site_available_languages: [en, en-us] behaviour. Sharded indexes
   where all children share one language (mcdonalds-* = 200-1000
   shards) pass through unchanged.

   No watermark state, no API call, no hybrid flag. Phenom's API has
   (a) no per-job timestamp, (b) no sort-by-recency, (c) Akamai-gated
   TLS/JS fingerprint needing a real Chrome profile — so the
   Eightfold watermark-on-postedTs pattern doesn't apply. Sitemap +
   _DIFF_BATCH + _MARK_GONE_BY_TIMESTAMP already provide the exact
   new/relisted/gone semantics from the shared pipeline.

Live-verified on Hetzner worker (2026-04-23) across all 9 tenants:

    tenant           before   after    api totalJob
    marriott           500   12,454       12,454
    nike               500      919          920
    nordstrom          500    1,105        1,107
    elevance-health    321      321          321
    mondelez           464      464          464
    nationwide         151      151          151
    mcdonalds-au       500    1,899        1,899
    mcdonalds-canada   500    7,938        3,889 (*)
    mcdonalds-us       500   37,496       53,844 (**)

(*) sitemap carries more URLs than the API's active filter; excess
    will age out via MARK_GONE_BY_TIMESTAMP if not re-seen.
(**) ~93/1000 shards carry French/Spanish suffixes filtered by the
     English-language gate; English coverage is the intent.

Tests: 28 new in test_phenom_monitor.py covering URL classification,
child-language extraction, per-language vs. sharded index selection,
and can_handle detection. 3,380 existing tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Bump crawler version to 0.8.75

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: viktor-shcherb

apps/crawler/VERSION

@@ -1 +1 @@
-0.8.74
+0.8.75

apps/crawler/data/boards.csv

@@ -441,6 +441,12 @@ def _build_comment(name: str, metadata: dict) -> str:
         if jobs is not None:
             return f"Personio XML \u2014 slug: {slug}, {jobs} jobs"
         return f"Personio XML \u2014 slug: {slug}"
+    if name == "phenom":
+        sitemap_url = metadata.get("sitemap_url", "?")
+        jobs = metadata.get("jobs")
+        if jobs is not None:
+            return f"Phenom \u2014 {jobs} jobs at {sitemap_url}"
+        return f"Phenom \u2014 {sitemap_url}"
     if name == "jobylon":
         group = metadata.get("company_group_id")
         company = metadata.get("company_id")
@@ -558,6 +564,7 @@ async def _probe_one(monitor: MonitorType) -> tuple[str, dict | None, str]:
     notion,  # noqa: F401
     oracle_hcm,  # noqa: F401
     personio,  # noqa: F401
+    phenom,  # noqa: F401
     pinpoint,  # noqa: F401
     recruitee,  # noqa: F401
     recruiter_co_kr,  # noqa: F401

apps/crawler/src/core/monitors/__init__.py

@@ -0,0 +1,247 @@
+"""Phenom People careers-site monitor.
+
+Phenom is a SaaS careers platform (phenompeople.com) used by large
+enterprises (Marriott, Nike, Nordstrom, Elevance, McDonald's, etc.).
+Every tenant exposes a sitemap at ``/sitemap.xml`` that is a sitemap-index
+pointing to child sitemaps. Child naming is either:
+
+- **Per-language** — one child per supported locale, each emitting a
+  distinct URL path per job (Marriott ``/ar/<slug>/job/<hex>__ar``,
+  Nike ``/de/<slug>/job/R-xxxxx``). Union-of-all-children explodes the
+  URL count by N languages; we keep only ``-en`` / ``-en-us`` children
+  to match the pre-existing ``site_available_languages: [en, en-us]``
+  behaviour of the old api_sniffer config.
+- **Sharded** — many children, all the same language suffix, each
+  carrying a subset of URLs (mcdonalds-au 210 shards, mcdonalds-canada
+  471 shards, mcdonalds-us 1000 shards). No per-language filtering is
+  applied; every shard contributes to the union.
+
+The sitemap is the authoritative URL set for gone detection; rich data
+for each job is extracted from the detail page's JSON-LD ``JobPosting``
+by the pipeline's ``json-ld`` scraper.
+
+Why not the ``/api/get-jobs`` endpoint? Phenom's API returns rich rows
+but has (a) no per-job timestamp, (b) no sort-by-recency, (c) Akamai-
+gated TLS/JS fingerprint that requires a real Chrome profile. That
+combination means the only way to get new-since-last-cycle from the
+API is a full paginated crawl of every tenant every cycle — 20 min
+for Marriott, 90 min for mchire. Sitemap plus per-URL json-ld is
+linear in *new URLs*, not total URLs, so steady-state is cheap.
+
+Incremental semantics come for free from the shared pipeline:
+``_DIFF_BATCH`` classifies each sitemap URL as new/relisted/touched
+against ``job_posting.last_seen_at``; ``_MARK_GONE_BY_TIMESTAMP``
+retires any active row whose URL wasn't re-seen this cycle. No
+watermark state, no hybrid flag, no API-sort assumption.
+
+Tenants migrated (2026-04-23): marriott, nike, nordstrom,
+elevance-health, nationwide, mondelez, mcdonalds-au, mcdonalds-canada,
+mcdonalds-us.
+"""
+
+from __future__ import annotations
+
+import re
+from urllib.parse import urlparse
+
+import httpx
+import structlog
+
+from src.core.monitors import register
+from src.core.monitors.sitemap import (
+    MAX_URLS,
+    _extract_child_sitemaps,
+    _extract_urls,
+    _is_sitemap_index,
+    _try_fetch_xml,
+)
+
+log = structlog.get_logger()
+
+# Phenom detail URLs contain either ``/job/<id>`` (canonical Phenom pattern,
+# e.g. ``careers.marriott.com/<slug>/job/<hex>``) or ``?job_id=<id>`` /
+# ``&job_id=<id>`` (mchire variant used by mcdonalds-us franchisees).
+# Case-insensitive because mchire renders ``/Job?job_id=...`` with a capital J.
+_JOB_URL_RE = re.compile(r"/job/|[?&]job_id=", re.IGNORECASE)
+
+# Phenom child sitemap filenames follow ``sitemap-<hex>-<lang>.xml`` (e.g.
+# ``sitemap-0a80f330-en.xml``). Used as the fingerprint in ``can_handle``.
+_PHENOM_CHILD_RE = re.compile(r"sitemap-[a-f0-9]+-[a-z-]+\.xml", re.IGNORECASE)
+
+# Languages we keep when the sitemap-index carries multiple locales. Matches
+# the old ``site_available_languages: [en, en-us]`` in the pre-migration
+# api_sniffer configs so the URL set stays equivalent.
+_KEEP_LANGS = frozenset({"en", "en-us"})
+
+
+def _is_phenom_job_url(url: str) -> bool:
+    return bool(_JOB_URL_RE.search(url))
+
+
+def _default_sitemap_url(board_url: str) -> str:
+    parsed = urlparse(board_url)
+    return f"{parsed.scheme}://{parsed.netloc}/sitemap.xml"
+
+
+def _child_language(child_url: str) -> str | None:
+    """Return the language suffix of a Phenom child sitemap filename, lowercased.
+
+    Phenom filenames follow ``sitemap-<hex>-<lang>[-<region>].xml``; the
+    language segment is everything after the second dash. Files with no
+    third segment (e.g. ``sitemap-content.xml`` at nationwide, which
+    holds only site-root URLs) return None — they get filtered out by
+    the job-URL regex downstream anyway.
+    """
+    name = child_url.rsplit("/", 1)[-1]
+    if not name.endswith(".xml"):
+        return None
+    parts = name[: -len(".xml")].split("-")
+    if len(parts) < 3:
+        return None
+    return "-".join(parts[2:]).lower()
+
+
+def _select_children(children: list[str]) -> list[str]:
+    """Filter child sitemap URLs to the English-equivalent subset.
+
+    Per-language indexes (marriott = 22 locales, nike = 16 locales) get
+    reduced to ``-en``/``-en-us`` children only. Sharded indexes where
+    every child carries the same language suffix (mcdonalds-*) pass
+    through unchanged because there is only one real language in the set.
+    Children without a language suffix are always kept — they're
+    typically low-cardinality content maps whose non-job URLs drop out
+    via ``_is_phenom_job_url``.
+    """
+    if not children:
+        return children
+    langs_with_suffix = {_child_language(c) for c in children} - {None}
+    # Only one real language → sharded layout, keep everything.
+    if len(langs_with_suffix) <= 1:
+        return children
+    return [c for c in children if _child_language(c) is None or _child_language(c) in _KEEP_LANGS]
+
+
+async def _collect_urls(
+    sitemap_url: str,
+    client: httpx.AsyncClient,
+) -> tuple[set[str], bool]:
+    """Fetch sitemap (index or flat), traverse selectively, return URL set.
+
+    Returns ``(urls, truncated)`` where *truncated* is True when we hit
+    ``sitemap.MAX_URLS`` during accumulation; the caller may log this.
+    """
+    root = await _try_fetch_xml(sitemap_url, client)
+    if root is None:
+        return set(), False
+
+    if not _is_sitemap_index(root):
+        return set(_extract_urls(root)), False
+
+    children = _extract_child_sitemaps(root)
+    selected = _select_children(children)
+    skipped = len(children) - len(selected)
+    if skipped:
+        log.debug(
+            "phenom.children_filtered",
+            sitemap=sitemap_url,
+            total=len(children),
+            kept=len(selected),
+            skipped=skipped,
+        )
+
+    urls: set[str] = set()
+    truncated = False
+    for child_url in selected:
+        if len(urls) >= MAX_URLS:
+            truncated = True
+            break
+        child_root = await _try_fetch_xml(child_url, client)
+        if child_root is None:
+            continue
+        # Nested sitemap-index (rare; defensive): single-level recurse.
+        if _is_sitemap_index(child_root):
+            for grandchild in _select_children(_extract_child_sitemaps(child_root)):
+                if len(urls) >= MAX_URLS:
+                    truncated = True
+                    break
+                gc_root = await _try_fetch_xml(grandchild, client)
+                if gc_root is not None:
+                    urls.update(_extract_urls(gc_root))
+        else:
+            urls.update(_extract_urls(child_root))
+    return urls, truncated
+
+
+async def discover(
+    board: dict,
+    client: httpx.AsyncClient,
+    pw=None,
+) -> tuple[set[str], str | None]:
+    """Fetch the Phenom sitemap, return (job_urls, new_sitemap_url).
+
+    Derives ``/sitemap.xml`` from the board host when not cached in
+    metadata, traverses the index (preferring English children for
+    per-language layouts), and filters the result to URLs that look
+    like job detail pages (``/job/`` or ``?job_id=``).
+
+    ``new_sitemap_url`` is non-None only when the monitor had to derive
+    the URL (mirrors the contract of ``sitemap.discover``). For boards
+    that already cache it in metadata, None is returned so the pipeline
+    skips the metadata-update write.
+    """
+    metadata = board.get("metadata") or {}
+    cached = metadata.get("sitemap_url")
+    sitemap_url = cached or _default_sitemap_url(board["board_url"])
+    new_sitemap_url = None if cached else sitemap_url
+
+    urls, truncated = await _collect_urls(sitemap_url, client)
+    job_urls = {u for u in urls if _is_phenom_job_url(u)}
+
+    log_fn = log.warning if truncated else log.info
+    log_fn(
+        "phenom.discover",
+        board_id=board.get("id"),
+        sitemap_urls=len(urls),
+        job_urls=len(job_urls),
+        truncated=truncated,
+    )
+    return job_urls, new_sitemap_url
+
+
+async def can_handle(
+    url: str,
+    client: httpx.AsyncClient | None = None,
+    pw=None,
+) -> dict | None:
+    """Detect Phenom by fingerprinting the sitemap-index child naming.
+
+    Every Phenom tenant's ``/sitemap.xml`` is a sitemap-index whose
+    children follow ``sitemap-<hex>-<lang>.xml``. Non-Phenom sitemaps
+    (static generators, WordPress, etc.) use different conventions, so
+    matching a single child filename is sufficient signal.
+
+    We avoid probing ``/api/get-jobs`` because Akamai returns 403 to
+    datacenter IPs (the probe can_handle runs from) even for valid
+    Phenom tenants — a 403 would be indistinguishable from a non-
+    Phenom host blocking the endpoint. The sitemap fingerprint is
+    cheaper and does not require egress proxying.
+    """
+    if client is None:
+        return None
+    parsed = urlparse(url)
+    sitemap = f"{parsed.scheme}://{parsed.netloc}/sitemap.xml"
+    root = await _try_fetch_xml(sitemap, client)
+    if root is None:
+        return None
+    children = _extract_child_sitemaps(root)
+    if not any(_PHENOM_CHILD_RE.search(c) for c in children):
+        return None
+    urls, _truncated = await _collect_urls(sitemap, client)
+    job_urls = sum(1 for u in urls if _is_phenom_job_url(u))
+    return {"sitemap_url": sitemap, "urls": len(urls), "jobs": job_urls}
+
+
+# Cost between eightfold (8) and sitemap (50) — same band as other
+# dedicated ATS monitors — so ``detect_monitor_type`` tries the Phenom
+# fingerprint before the generic sitemap path for new Phenom tenants.
+register("phenom", discover, cost=9, can_handle=can_handle)

apps/crawler/src/core/monitors/phenom.py

@@ -97,8 +97,11 @@ async def _fetch(method: str, url: str, headers: dict, body: str | None) -> obje
 )
 
 COUNT_FIELDS = re.compile(
+    # `total[A-Za-z]+` covers vendor-specific suffixes like Phenom's `totalJob`
+    # (job siblings to the array are always int-valued and evaluated in context,
+    # so the broader match stays anchored to the array's parent object).
     r"^(total|count|total_?count|total_?results|total_?items|hits|num_?found|result_?count"
-    r"|size|totalCount|totalResults|totalItems|totalHits|nbHits)$",
+    r"|size|total[A-Za-z]+|nbHits)$",
     re.IGNORECASE,
 )
 

apps/crawler/src/shared/api_sniff.py

@@ -66,6 +66,7 @@ def auto_skip_crawler_types() -> frozenset[str]:
     "workday",
     "ycombinator",
     "sitemap",
+    "phenom",
     "nextdata",
     "notion",
     "dom",
@@ -254,6 +255,11 @@ def auto_scraper_type(
         return ("workday", None)
     if monitor_type == "eightfold":
         return ("eightfold", None)
+    if monitor_type == "phenom":
+        # Every tenant has a JSON-LD JobPosting on the detail page.
+        # Tenants whose pages need Playwright render (mcdonalds-*, nationwide
+        # detail) override with ``{"render": true}`` in boards.csv.
+        return ("json-ld", None)
     if monitor_type == "softgarden":
         return ("json-ld", None)
     if monitor_type == "ycombinator":

apps/crawler/src/workspace/_compat.py

@@ -606,6 +606,45 @@
 
   Pair with:     json-ld (try first) or dom scraper"""
 
+MONITOR_PHENOM = """\
+phenom — Phenom People Careers Platform (sitemap + json-ld)
+
+  Returns:  URL set only (needs scraper)
+  Cap:      50,000 URLs (inherited from sitemap)
+
+  Phenom tenants (e.g. careers.marriott.com, careers.nike.com) expose a
+  sitemap-index at /sitemap.xml with per-language child sitemaps named
+  sitemap-<hex>-<lang>.xml. The sitemap is the authoritative URL set.
+  Rich job data comes from the detail page's JSON-LD JobPosting script,
+  extracted by the json-ld scraper.
+
+  Why a dedicated monitor vs. plain sitemap:
+    • Phenom-specific can_handle fingerprint (child naming) avoids
+      mis-detecting generic XML sitemaps as Phenom during ws probe.
+    • Filters discovered URLs to job detail pages (contain "/job/" or
+      "?job_id="), dropping site root, "/jobs" index, language pages.
+
+  No API is used. Phenom's /api/get-jobs has no per-job timestamp and
+  no sort-by-recency, so there's no meaningful incremental signal; the
+  sitemap URL set + last_seen_at in Postgres already give _DIFF_BATCH
+  everything it needs for new/relisted/gone classification.
+
+  Config:
+    {}  — no configuration required. sitemap_url is derived from the
+          board URL's scheme+host (board metadata stores the discovered
+          URL for future runs, as with the sitemap monitor).
+
+  Scrapers:
+    json-ld (default) — works for marriott, nike, nordstrom, elevance,
+                        mondelez. Detail page returns JSON-LD natively.
+    json-ld + render:true — for mcdonalds-* and nationwide detail pages
+                            where Playwright render is needed before
+                            the JobPosting <script> appears in DOM.
+
+  Browser flags on the board (persistent_context, channel=chrome,
+  proxy) only matter for the scraper path; the monitor itself uses
+  plain httpx and runs in ~5 seconds regardless of job count."""
+
 MONITOR_NEXTDATA = """\
 nextdata — Next.js __NEXT_DATA__ Discovery
 
@@ -1945,6 +1984,7 @@
     "personio": MONITOR_PERSONIO,
     "rss": MONITOR_RSS,
     "sitemap": MONITOR_SITEMAP,
+    "phenom": MONITOR_PHENOM,
     "nextdata": MONITOR_NEXTDATA,
     "notion": MONITOR_NOTION,
     "oracle_hcm": """\

apps/crawler/src/workspace/commands/help.py

@@ -181,6 +181,10 @@ def test_total_items_nested(self):
         body = {"data": {"totalItems": 42, "jobs": [{"id": 1}]}}
         assert find_total_count(body, "data.jobs") == 42
 
+    def test_total_job_phenom(self):
+        body = {"totalJob": 12441, "jobs": [{"id": 1}], "facets": []}
+        assert find_total_count(body, "jobs") == 12441
+
 
 class TestScoreCandidate:
     def test_high_score_with_url_and_title(self):