Skip to content

Commit 5917e68

Browse files
author
David
committed
Bump transfermarkt-scraper to latest main and fix clubs merge
Update the pinned transfermarkt-scraper git dependency to the latest main commit, which brings a crawlee 1.6.0 -> 1.9.0 bump. The new scraper version's clubs output made parent.average_market_value type as JSON instead of VARCHAR in some scrapes, breaking the DuckDB UNION ALL BY NAME merge with existing raw data. Add 'clubs' to JSON_CASTS to normalize the parent struct, matching the same treatment already applied to players, games, game_lineups, appearances, and national_team_players.
1 parent af8c44a commit 5917e68

2 files changed

Lines changed: 16 additions & 13 deletions

File tree

poetry.lock

Lines changed: 15 additions & 13 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

scripts/acquiring/transfermarkt-scraper.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ def all(cls):
109109
# Casting to JSON normalises the type and avoids UNION ALL BY NAME conflicts.
110110
# Safe to do because base models only ever access these via json_extract_string().
111111
JSON_CASTS = {
112+
'clubs': ['parent'],
112113
'players': ['parent'],
113114
'games': ['parent'],
114115
'game_lineups': ['parent'],

0 commit comments

Comments
 (0)