Commit 179e43e
* Add AlmaCareer monitor for jobs.cz (CZ) and topjobs.sk (SK)
Alma Career runs the Capybara-framework career sites used by many
employers in Czechia and Slovakia. Both countries share one GraphQL
backend at api.capybara.lmc.cz — only the host, widgetId, and
per-tenant apiKey differ, so they are handled by a single `almacareer`
monitor type with a `country` metadata field.
The monitor:
- Extracts widgetId + apiKey from the tenant's /assets/js/script.min.js
bundle (they are baked in at deploy time).
- Walks `jobAdList` pagination (server-capped at 10 items/page).
- Fires per-job `jobAd(id)` queries concurrently to pull full
content.htmlContent (the list payload only includes a teaser).
- Registers as a rich monitor (scraper auto-skipped).
Closes #2434.
* Bump apps/crawler/VERSION to 0.8.65
Covers the AlmaCareer monitor addition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Apply Stage 2 review polish + add 10 pilot companies
Polish (apps/crawler/src/core/monitors/almacareer.py):
- Harden _WIDGET_MAIN_RE: switch from balanced-brace body match to
anchor-on-"widgets":{"main":{ + 4 KB bounded window; tolerant to nested
objects ("themes", "filters") between "id" and "apiKey" in the minified
bundle.
- GraphQL partial-data handling: prefer payload.data when both errors and
data are present (Capybara routinely returns both for optional-field
resolver failures); log "graphql_partial_data" and only raise if data
is missing.
- Log _fetch_widget_config exceptions and non-200 responses under
"widget_config_fetch_failed" instead of swallowing silently.
- Loosen _parse_location: keep up to five distinct parts (cityPart, city,
district, region, country) instead of silently dropping district+region
past two leading parts.
- gather(return_exceptions=True) on the detail fan-out so one flaky
detail fetch doesn't poison the whole discover run; log
"hydrate_failed" per-job.
Tests (apps/crawler/tests/test_almacareer.py, +6 cases):
- Empty listing returns [] without firing detail fetches.
- _fetch_job_html returning None preserves the listing teaser.
- can_handle page-HTML marker fallback (custom-domain portal finds tenant
via cdn.capybara.lmc.cz + data-host).
- _match_country uppercase host.
- _parse_location with five distinct parts.
- _extract_widget_config with nested objects before "apiKey".
Pilot companies (10 boards via the new monitor):
- New CZ companies: albert-cz, csob, kaufland-cz, kb-cz, o2-cz.
- Additional boards on existing companies: bosch, ey, kpmg, siemens,
vodafone (each gets a *-cz-almacareer board).
All 10 pilots verified with direct discover() probe (alma GraphQL only):
albert 173, kaufland 139, csob 107, o2 94, vodafone 90, kb 71, ey 45,
siemens 43, bosch 31, kpmg 23 — total 816 jobs.
apps/crawler/VERSION: 0.8.65 → 0.8.70.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fill full metadata + R2-staged logos for 5 new CZ companies
* Upload images to R2
* AlmaCareer: hardening + 8 more pilots (5 CZ + 2 existing + 1 SK)
Review findings from a close read of the monitor + `can_handle` path,
plus eight additional live-verified pilots including the first
`topjobs.sk` tenant.
Code fixes
----------
- **``BoardGoneError`` on 404 `script.min.js`.** `_fetch_widget_config`
previously swallowed every non-200 to `None`, which raised a generic
`ValueError` in `discover` — the board would retry 5 times before
auto-disabling. Now 404 → `_WidgetConfigGone` → `BoardGoneError`
(one-shot). Transport errors propagate as retriable.
- **`_WIDGET_WINDOW_BYTES`: 4KB → 64KB.** The bundle is <300KB; the
old window had thin margin if Alma ever enlarges `themes` / `filters`
between the `main` anchor and `apiKey`. Single-pass regex; linear
in window size.
- **`null` `jobAdList` raises.** GraphQL schema drift would otherwise
surface as a silent 0-job crawl that eventually auto-disables the
board. Now it raises so the failure is loud.
- **`o2-cz` logo_type: `lettermark` → `wordmark+icon`.** `lettermark`
wasn't in the validator's enum, so the PR shipped with a dormant
validation error that `inspect.validate_csvs()` now surfaces.
Pilots (live-verified from my IP 2026-04-23)
--------------------------------------------
Existing companies (new boards):
- `mcdonalds-cz-almacareer` mcdonalds.jobs.cz 169 jobs
- `mcdonalds-sk-almacareer` mcdonalds.topjobs.sk 18 jobs (first SK tenant)
- `bmw-cz-almacareer` bmw.jobs.cz 26 jobs
New companies (wikidataId + SVG logo + icon + en description):
- `penny-cz` Penny Market 273 jobs
- `kfc` Kentucky Fried Chicken 180 jobs
- `billa-cz` Billa 155 jobs
- `t-mobile-cz` T-Mobile Czech 61 jobs
- `ikea-cz` IKEA CZ 9 jobs
Rewe-group sibling retailers (Penny, Billa) carry a shared
``parentOrganization`` note; KFC is Yum! Brands; T-Mobile CZ is
Deutsche Telekom.
All logos sourced from Wikimedia Commons (SVG) + apple-touch-icons /
favicons from each tenant's home page.
Addresses critic's N2 (no SK pilot verified live) by including
`mcdonalds-sk-almacareer`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent cf9ab18 commit 179e43e
21 files changed
Lines changed: 2131 additions & 82 deletions
File tree
- apps/crawler
- data
- images
- billa-cz
- ikea-cz
- kfc
- penny-cz
- t-mobile-cz
- src
- core/monitors
- workspace
- commands
- tests
- scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Loading
Loading
Binary file not shown.
Loading
Binary file not shown.
0 commit comments