Skip to content

Commit 179e43e

Browse files
viktor-shcherbclaudegithub-actions[bot]
authored
AlmaCareer monitor + 18 pilots (supersedes #2443) (#2540)
* Add AlmaCareer monitor for jobs.cz (CZ) and topjobs.sk (SK) Alma Career runs the Capybara-framework career sites used by many employers in Czechia and Slovakia. Both countries share one GraphQL backend at api.capybara.lmc.cz — only the host, widgetId, and per-tenant apiKey differ, so they are handled by a single `almacareer` monitor type with a `country` metadata field. The monitor: - Extracts widgetId + apiKey from the tenant's /assets/js/script.min.js bundle (they are baked in at deploy time). - Walks `jobAdList` pagination (server-capped at 10 items/page). - Fires per-job `jobAd(id)` queries concurrently to pull full content.htmlContent (the list payload only includes a teaser). - Registers as a rich monitor (scraper auto-skipped). Closes #2434. * Bump apps/crawler/VERSION to 0.8.65 Covers the AlmaCareer monitor addition. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Apply Stage 2 review polish + add 10 pilot companies Polish (apps/crawler/src/core/monitors/almacareer.py): - Harden _WIDGET_MAIN_RE: switch from balanced-brace body match to anchor-on-"widgets":{"main":{ + 4 KB bounded window; tolerant to nested objects ("themes", "filters") between "id" and "apiKey" in the minified bundle. - GraphQL partial-data handling: prefer payload.data when both errors and data are present (Capybara routinely returns both for optional-field resolver failures); log "graphql_partial_data" and only raise if data is missing. - Log _fetch_widget_config exceptions and non-200 responses under "widget_config_fetch_failed" instead of swallowing silently. - Loosen _parse_location: keep up to five distinct parts (cityPart, city, district, region, country) instead of silently dropping district+region past two leading parts. - gather(return_exceptions=True) on the detail fan-out so one flaky detail fetch doesn't poison the whole discover run; log "hydrate_failed" per-job. Tests (apps/crawler/tests/test_almacareer.py, +6 cases): - Empty listing returns [] without firing detail fetches. - _fetch_job_html returning None preserves the listing teaser. - can_handle page-HTML marker fallback (custom-domain portal finds tenant via cdn.capybara.lmc.cz + data-host). - _match_country uppercase host. - _parse_location with five distinct parts. - _extract_widget_config with nested objects before "apiKey". Pilot companies (10 boards via the new monitor): - New CZ companies: albert-cz, csob, kaufland-cz, kb-cz, o2-cz. - Additional boards on existing companies: bosch, ey, kpmg, siemens, vodafone (each gets a *-cz-almacareer board). All 10 pilots verified with direct discover() probe (alma GraphQL only): albert 173, kaufland 139, csob 107, o2 94, vodafone 90, kb 71, ey 45, siemens 43, bosch 31, kpmg 23 — total 816 jobs. apps/crawler/VERSION: 0.8.65 → 0.8.70. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fill full metadata + R2-staged logos for 5 new CZ companies * Upload images to R2 * AlmaCareer: hardening + 8 more pilots (5 CZ + 2 existing + 1 SK) Review findings from a close read of the monitor + `can_handle` path, plus eight additional live-verified pilots including the first `topjobs.sk` tenant. Code fixes ---------- - **``BoardGoneError`` on 404 `script.min.js`.** `_fetch_widget_config` previously swallowed every non-200 to `None`, which raised a generic `ValueError` in `discover` — the board would retry 5 times before auto-disabling. Now 404 → `_WidgetConfigGone` → `BoardGoneError` (one-shot). Transport errors propagate as retriable. - **`_WIDGET_WINDOW_BYTES`: 4KB → 64KB.** The bundle is <300KB; the old window had thin margin if Alma ever enlarges `themes` / `filters` between the `main` anchor and `apiKey`. Single-pass regex; linear in window size. - **`null` `jobAdList` raises.** GraphQL schema drift would otherwise surface as a silent 0-job crawl that eventually auto-disables the board. Now it raises so the failure is loud. - **`o2-cz` logo_type: `lettermark` → `wordmark+icon`.** `lettermark` wasn't in the validator's enum, so the PR shipped with a dormant validation error that `inspect.validate_csvs()` now surfaces. Pilots (live-verified from my IP 2026-04-23) -------------------------------------------- Existing companies (new boards): - `mcdonalds-cz-almacareer` mcdonalds.jobs.cz 169 jobs - `mcdonalds-sk-almacareer` mcdonalds.topjobs.sk 18 jobs (first SK tenant) - `bmw-cz-almacareer` bmw.jobs.cz 26 jobs New companies (wikidataId + SVG logo + icon + en description): - `penny-cz` Penny Market 273 jobs - `kfc` Kentucky Fried Chicken 180 jobs - `billa-cz` Billa 155 jobs - `t-mobile-cz` T-Mobile Czech 61 jobs - `ikea-cz` IKEA CZ 9 jobs Rewe-group sibling retailers (Penny, Billa) carry a shared ``parentOrganization`` note; KFC is Yum! Brands; T-Mobile CZ is Deutsche Telekom. All logos sourced from Wikimedia Commons (SVG) + apple-touch-icons / favicons from each tenant's home page. Addresses critic's N2 (no SK pilot verified live) by including `mcdonalds-sk-almacareer`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent cf9ab18 commit 179e43e

21 files changed

Lines changed: 2131 additions & 82 deletions

File tree

VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.7.4
1+
0.7.5

apps/crawler/VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.8.75
1+
0.8.76

apps/crawler/data/boards.csv

Lines changed: 32 additions & 0 deletions
Large diffs are not rendered by default.

apps/crawler/data/companies.csv

Lines changed: 10 additions & 0 deletions
Large diffs are not rendered by default.

apps/crawler/data/company_descriptions.csv

Lines changed: 5 additions & 80 deletions
Large diffs are not rendered by default.
3.65 KB
Loading
Lines changed: 19 additions & 0 deletions
Loading
7.23 KB
Binary file not shown.
Lines changed: 40 additions & 0 deletions
Loading
10.7 KB
Binary file not shown.

0 commit comments

Comments
 (0)