This project downloads and archives "Proces verbaal" (vote counting) documents published by Dutch municipalities after elections.
Any text generated by an agent for this repository should be written in English, even when the source material, municipality website, or copied notes are in Dutch. This includes README updates, TODO notes, config comments, and other agent-authored documentation.
{election}/ # Election folder (e.g. 2026-GR)
TODO.md # Problematic municipalities only
{code}/ # CBS municipality code
.lock # Lock file while processing
.todo # URL for municipality (one line; when no SHA256SUMS)
config.txt # Download configuration
README.md # Municipality-specific notes
*.pdf # Downloaded documents
SHA256SUMS # File checksums
When a new election occurs:
-
Create the election folder
mkdir {election} -
Create the election README.md Include the Kiesraad results page URL:
# {year}-{type} {election name} - Election date: {date} - Kiesraad results: {url}
-
Initialize TODO.md (problems only) Start with just the header (and the retention note if needed). Do not populate it with all municipalities. Add entries only when a municipality is blocked or needs follow-up.
-
Scrape the municipality list from the Kiesraad page (once per election cycle) Run this once to produce
progress/{election}-municipalities.txt:./scripts/scrape_kiesraad_municipalities.py 2026-GR <kiesraad_url>
This creates a TSV file with
code\tname\tkiesraad_urlfor every municipality. The file lives inprogress/which is gitignored. Do not pre-create empty municipality directories; directories are created on demand when a municipality is actually processed. -
Copy configs from previous election (on demand) When processing a municipality, copy
config.txtandREADME.mdfrom the previous election if they exist. Do this per-municipality, not in bulk:mkdir -p {election}/{code} cp {prev_election}/{code}/config.txt {election}/{code}/ 2>/dev/null || true cp {prev_election}/{code}/README.md {election}/{code}/ 2>/dev/null || trueDo NOT copy PDF files, URL lists, or SHA256SUMS.
{election}/TODO.md only lists problematic municipalities. Each entry is a ## {code} {name} heading followed by notes. Do not include URL lines in TODO.md.
- Store the municipality URL in
{election}/{code}/.todo(single-line file) for any municipality withoutSHA256SUMS(not just problematic ones). - For entries in TODO.md,
.todomust exist and contain the URL. - When a municipality is completed and
SHA256SUMSexists, remove the.todofile during main-agent finalization. - A municipality not in TODO.md may be unprocessed or already completed; use
.lockandSHA256SUMSfor progress tracking. - URLs are stored in
progress/{election}-municipalities.txt(produced byscrape_kiesraad_municipalities.py). For quick lookup of a single municipality's Kiesraad URL, grep that file. SHA256SUMSmust never include.lockor.todoentries; if it does, regenerate after removing those files.- Prefer editor/file tools to delete
.lockand.todoso subagents do not trigger avoidable shell approval prompts. Usermonly when you are already in the terminal for a broader cleanup. - Do not hand-edit
SHA256SUMS; regenerate via./scripts/fetch-pv.pyif needed. Important: SHA256SUMS files are sorted by filename (2nd field) for reproducibility — ensure any manual regeneration usessort -k 2to keep output deterministic. - For bulk cleanup of generated municipality files (
config.txt, URL list, PDFs,SHA256SUMS, and optionally.todo), prefer./scripts/cleanup-municipality.pyover longrmcommands with many arguments or globs.
Municipalities are legally required to retain processen-verbaal for 3 months after an election. After that, they may delete them.
When searching for processen-verbaal:
- < 2 weeks after election: Files may not be uploaded yet. Note in TODO and retry later.
- 2 weeks – 3 months after election: Files should be available. If not found, note the URLs searched in TODO.
- > 3 months after election: Files may have been deleted. Note "likely deleted" in TODO and move on.
When processing multiple municipalities, handle them one at a time (not in a loop). This makes it easier to:
- Diagnose failures for individual municipalities
- Allow the user to follow along
- Track progress via the TODO list
Finish with a summary: After processing multiple municipalities, end your response with a brief summary of what completed, what was partial/blocked, and any follow-up needed. Use a table with one row per municipality.
Sanity check: Do not run any bulk repair or URL-population script after each batch. These are one-off tools and should only be used when explicitly asked.
Quick progress check: Run ./scripts/show-progress.py --election {election} anytime to display the current finished, in-progress, and TODO counts. When progress/{election}-municipalities.txt exists, the total comes from there (not from directory count).
In addition to {election}/TODO.md, maintain a separate agent todo list (the assistant task list) so progress is visible at a glance.
Requirements:
- Create the todo list before starting a batch.
- Add one item per municipality in the batch (use the CBS code and name).
- Mark each item as completed immediately after finishing that municipality.
- If a municipality is blocked/partial, still mark the item completed and note the reason in the todo item text.
- Do not end the session with all items still marked in-progress.
When using subagents, prefer one municipality per subagent. This keeps locking, README updates, TODO tracking, verification, and commits isolated and easier to review.
- Default to 3 subagents in parallel.
- Increase beyond 3 only if the user explicitly wants it and permission prompts / model rate limits are not causing trouble.
- Avoid large batches per subagent unless several municipalities clearly use the same platform and the user asked for that trade-off.
- When launching multiple subagents, start them in one parallel tool call so they actually run concurrently. Do not invoke
runSubagentsequentially in separate calls when the intent is parallel execution. - When the user explicitly wants a larger batch, a proven pattern is 15 municipalities split across 5 subagents (3 municipalities each), all launched in that single parallel call.
- Name each
runSubagentdescription after the municipalities it contains, e.g."Borger-Odoorn, Oegstgeest, Weststellingwerf"rather than a generic label like"Process 3 municipalities 2026-GR". This makes progress visible in the UI.
Continuous refill (ideal): When a subagent finishes, the main agent should
immediately select 1 more municipality and launch a replacement subagent, keeping
the pipeline full. In practice the runSubagent tool is synchronous: parallel
subagents all return at the same time, so true continuous refill is not possible.
Keep batch sizes small (3) to minimise idle time.
Subagent responsibilities:
- Work only in the assigned municipality directory unless there is a clear reason not to.
- If a subagent is assigned multiple municipalities, finish, finalize, and commit each municipality immediately before starting the next one. Do not wait and commit the whole subagent batch at the end.
- Do not edit
{election}/TODO.mddirectly; return the exact TODO block text to the main agent when a municipality is partial or blocked. - Verify the result before finishing.
- If a dedicated fetch script (for example
scrapers/mijnstembureau.py) fails unexpectedly or appears to require code changes, stop and hand the problem back to the main agent. Do not spend subagent time debugging shared scripts. - Commits: Subagents should commit their own successful municipality work (see commit rules below). If the commit fails or anything unexpected happens, do not retry or amend — hand back to the main agent with the exact error.
- Finalization on success: Remove
.lockand.todo(prefer editor/file tools overrm), then commit the metadata files. Return a concise status summary to the main agent. - On failure/partial: Leave
.lockand.todoin place and return the status to the main agent for cleanup and TODO.md updates.
Never commit PLAN, PROGRESS, REPORT, COMPLETE, or similar working/planning files. These are temporary agent notes and belong in tmp/ (which is gitignored), not in the main repository. When generating summary reports or progress tracking documents:
- Save to
tmp/{filename}.mdso they're excluded from git - Reference them in chat as needed but document the findings in proper locations (README.md, TODO.md, or config files where applicable)
- Delete stale planning files from
tmp/when no longer needed
This keeps the repository clean and focused on actual data (municipality directories, PDFs, config files, and documentation).
Commit rules (subagents and main agent):
- Keep commits strictly scoped to one municipality.
- If a subagent is working through multiple municipalities, create the commit for municipality A before starting municipality B. Never hold several completed municipalities for one end-of-batch commit.
- Stage an explicit file list:
config.txt,README.md,SHA256SUMS, and the generated URL list ({code} {name}.txt). - Never use
git add -f {election}/{code}or stage a whole municipality directory blindly: some downloaded files do not end in.pdf, so whole-directory staging can accidentally commit assets. - Never use
git commit --amend. If the staged set is wrong, hand back to the main agent. - A success commit message is just
{code} {name}. - A partial/blocked municipality may also need
{election}/TODO.md, but verify the diff contains only that municipality's TODO block. - Keep workflow/documentation changes (such as
AGENTS.md) in a separate commit from municipality results.
The selection script reads from progress/{election}-municipalities.txt (produced once
by scrape_kiesraad_municipalities.py). It excludes municipalities that already have
a directory with SHA256SUMS, a .lock, or an entry in TODO.md.
./scripts/select_random_municipalities.py 2026-GR 6Output is TSV: code\tname\tkiesraad_url. Use the URL from the output to seed .todo.
Note: If other agents have already locked all remaining municipalities (or all remaining ones are already in TODO.md), the selection script will report “No eligible municipalities found.” In that case, ask the user how to proceed. If it returns fewer than N results, that’s fine, just work on the smaller set.
Before starting work, lock all chosen municipalities by creating a .lock file in each directory:
mkdir -p {election}/{code} && touch {election}/{code}/.lockSafe TODO.md updates (concurrent edits):
- Re-read the latest TODO.md immediately before each edit.
- Edit only the specific municipality block you are working on (avoid broad replacements or reformatting).
- If the block has changed (notes added, removed, or the entry disappeared), do not overwrite it; re-select another municipality.
- TODO.md should contain notes only (no URL lines). URLs live in
.todofiles.
Process each municipality sequentially. After completing each municipality (before moving to the next):
- Ensure
{election}/{code}/.lockexists before starting. - Ensure
{election}/{code}/.todoexists and contains the URL before running fetches. - If successful: remove
.lockand.todo, commit that municipality's metadata files immediately, then report back or move to the next municipality. - If failed/partial: leave
.lockand.todoin place, add/update the TODO.md entry with notes (no URL lines), and hand back to the main agent. - Mark the todo item as completed in your todo list.
- Verify the download worked before moving to the next.
Important: Update TODO.md incrementally after each municipality, not in a batch at the end. This ensures progress is saved and other agents see accurate state.
Do not update the OTS timestamp after each municipality.
- Once an election is fully processed, the main agent can do the OTS update in one go.
- If only a few stubborn municipalities remain and the rest is complete, ask the user whether they want to do OTS earlier.
- Keep OTS work separate from municipality-processing commits when practical.
For each municipality:
-
Find the processen-verbaal page
- If
{election}/{code}/.todoexists, start from that URL - Otherwise, locate the URL via config or the municipality website
- Look for links like "Processen-verbaal", "Uitslag per stembureau", or similar
- Note: Some sites have PDFs directly on the main page, others have a dedicated subpage
- If
-
Create the municipality folder and config
mkdir -p {election}/{code}Create
config.txtwith:URL=<page containing PDF links> REGEX=<regex to match PDF links> PREFIX=<prefix for relative URLs> NAME=<municipality name> -
Run the fetch script
./scripts/fetch-pv.py {election}/{code}Do not call the venv binary directly (e.g.
.venv/bin/python) — it triggers permission prompts. This will:- Generate a URL list file (
{code} {name}.txt) if missing - Download all PDFs (skipping existing files)
- Generate
SHA256SUMSor verify existing checksums
- Generate a URL list file (
-
Sanity check against previous elections Compare the number of stembureaus with previous years using available election folders:
wc -l {prev_election}/{code}/*.txt {election}/{code}/*.txt- Some variation (±20%) is normal due to population changes
- Large drops (>50%) may indicate missing files or wrong regex
- File naming conventions may differ between years
-
Cross-verify against official counting data (osv4-3) Many municipalities publish a "tellingsbestand" CSV (osv4-3 format) on their open data portal. This file lists every stembureau with vote counts per candidate. Use it to verify completeness:
- Download the CSV (look for
osv4-3-telling-*.csvon the municipality's open data site) - Extract the stembureau numbers from the
"Gebiednummer"row - Compare against the stembureau numbers in the downloaded PV filenames
- They should match exactly (PDF count = stembureaus in CSV + summary PVs)
This is optional but recommended, especially for large municipalities.
- Download the CSV (look for
-
Update README.md with stembureau counts Create or update
README.mdin the municipality folder:# {name} ## Stembureaus per verkiezing | Verkiezing | Stembureaus | |------------|-------------| | 2023-TK | XX | | 2024-EP | XX | | 2025-TK | XX | | 2026-GR | XX |
Omit rows with a count of 0 (it means that election was not downloaded).
-
Verify and clear tracking
- Check the downloaded PDFs are correct proces-verbaal documents
- If only summary PDFs are found (no per-stembureau PVs), treat as partial: keep the TODO.md entry and
.todo, and remove anySHA256SUMScreated by the partial fetch - Remove the municipality from TODO.md (if present)
- On success, remove
{election}/{code}/.todoand.lock(prefer editor/file tools), then commit - On failure/partial, leave
.lockand.todoin place and hand back to the main agent - If you need to discard a bad partial fetch with many generated files, prefer
./scripts/cleanup-municipality.py {election}/{code} --reset-generatedover a longrmcommand
-
Commit
git add {election}/{code}/config.txt {election}/{code}/README.md {election}/{code}/SHA256SUMS '{election}/{code}/{code} {name}.txt' git commit -m "{code} {name}"For partial municipalities that also require a TODO update, carefully add only the municipality directory plus the relevant election
TODO.md.
| Municipality Type | REGEX Pattern | Notes |
|---|---|---|
| Standard PDFs | proces.*verbaal.*\.pdf |
Case-insensitive |
| TYPO3/Fileadmin | fileadmin/.*\.pdf |
Common CMS |
| Drupal | sites/.*/files/.*\.pdf |
Common CMS |
| sim-cdn.nl | sim-cdn\.nl/.*/uploads/.*\.pdf |
CDN hosting |
| SDU/dsresource | dsresource\?objectid=.*type=pdf |
See below |
Some municipalities use SIMsite with assets on cuatro.sim-cdn.nl. The HTML is a
Next.js page, but the documenten are often still discoverable without a browser:
- Check whether the page contains a
__NEXT_DATA__script payload. - Look in that JSON for a dedicated page such as
/processen-verbaal-gemeenteraadsverkiezing-2026(or a similar verkiezingen subpage) rather than assuming the generic/verkiezingenpage is the right one. - Repoint
config.txtto that dedicated page and use a regex likeachtkarspelen/uploads/.*\.(pdf|csv)or the municipality-specificsim-cdn\.nl/.../uploads/.*\.pdfvariant.
Do not assume a browser is required just because the site is Next.js. For
example, Achtkarspelen exposes all documenten directly in the __NEXT_DATA__
payload on the dedicated processen-verbaal page, and fetch-pv.py can download
them normally once URL is set correctly.
Some municipalities use TYPO3 pages with election documents living under
/fileadmin/verkiezingen/.... These pages often mix several kinds of downloads
on one page:
- station-level PVs, usually in a
processen-verbaal/subfolder - per-stembureau result attachments, often in
uitkomsten-per-stembureau/ - a central-bureau PV, sometimes alongside the station PVs
- a CSV sidecar with the digital telling
Veenendaal is a useful variant of this pattern: the same results page exposes
both *_eerste_telling.pdf files and the final station PDFs, plus the
central-bureau summary PDFs and a spreadsheet sidecar. In that case, a broad
\.pdf fetch is usually enough, and the stembureau count should still be based
on the unique station numbers rather than the total PDF count.
When this happens, use a broad fileadmin/.../gemeenteraadsverkiezingen-2026/.*\.(pdf|csv)
style regex, then count only the station-level PVs for the README stembureau
count. Gooise Meren is a good example: the page publishes the central counting
PDFs, the per-station PVs, and the CSV all in one place.
Some Drupal municipalities expose the archive on a separate public-documents
page rather than the main verkiezingen summary. Westerwolde is a good example:
the summary page points to /officiele-publicaties, and that page carries the
actual station-level PDFs, the osv4-3 CSV, and the central count documents.
For this pattern, a broad sites/default/files/2026-.*\.(pdf|csv) regex is
usually enough, and the README count should come from the station-level sb-*
documents rather than the total number of attachments.
Some municipalities use Open Online pages where the useful downloads live in the
__NEXT_DATA__ payload as attachment lists. These pages often have a dedicated
result route such as /verkiezingsuitslag-gemeenteraad-2026 even when the
generic /verkiezingen landing page looks sparse.
Look for attachment sections like:
Proces-verbaal centraal stembureauProces-verbaal gemeentelijk stembureauProcessen-verbaal stembureausUitslagen per stembureau
Kampen is a good example: the result page exposes the full set of stembureau
PVs, the 1e_telling PDFs, the central-bureau PV, and the digital CSV in one
__NEXT_DATA__ blob, so fetch-pv.py can download the archive once config.txt
points at the dedicated result page.
Some other municipal CMS pages expose a dedicated result subpage with links
ending in .org even though the response is actually a PDF or CSV. De Ronde
Venen and Oudewater are examples: point config.txt at the dedicated
Processen_verbaal_van_de_stembureaus or results page and let
fetch-pv.py resolve the downloaded filename from the response headers.
Some municipalities use SDU's CMS which serves files via URLs like:
https://www.gemeente.nl/dsresource?objectid=abc123-def456&type=pdf
The fetch-pv.py script handles this automatically by extracting the filename from the Content-Disposition header.
Example config:
URL=https://www.vlaardingen.nl/Bestuur/Verkiezingen/Tweede_Kamerverkiezing_2025
REGEX=dsresource\?objectid=.*type=pdf
PREFIX=https://www.vlaardingen.nl/
NAME=Vlaardingen
Some municipalities use "centraal tellen" (central counting) where votes from all polling stations are counted together at a central location the day after the election. In this case:
- There is typically one combined PDF for all stembureaus instead of individual files
- The stembureau count in README.md should note this:
1 (centraal tellen)or similar - The number of stembureaus will appear much lower than previous years
This is a valid approach and not an error - just document it in the README.md.
Some municipalities host files on StackStorage (e.g., technischbeheerassen.stackstorage.com). The share pages are JavaScript shells, but the underlying public-share API can be used directly without browser automation.
Use the dedicated script, passing one or more public share URLs:
./scripts/scrapers/stackstorage.py \
https://{disk}.stackstorage.com/s/{shareId} \
... \
{election}/{code}The script handles authentication (POST to obtain tokens), directory listing, and file download automatically, then generates a sorted SHA256SUMS.
Record the share URLs in config.txt comments and in the municipality URL list so the source remains reproducible. Look for live share URLs on the municipality page first — old ones may go stale.
If the script fails or the municipality only exposes private shares, stop and look for another static/API approach before considering browser automation.
Some municipalities use the "Mijn Stembureau" platform (e.g., mijnstembureau-{gemeente}.nl). These require a different approach:
-
Create config.txt with a note
# Uses Mijn Stembureau API # Download with: ./scripts/scrapers/mijnstembureau.py <url> <dir> NAME=Losser -
Run the dedicated script
./scripts/scrapers/mijnstembureau.py https://mijnstembureau-{gemeente}.nl {election}/{code} -
Add a note to README.md
# {name} Uses [Mijn Stembureau](https://mijnstembureau-{gemeente}.nl/) platform. Download with: ```bash ./scripts/scrapers/mijnstembureau.py https://mijnstembureau-{gemeente}.nl {election}/{code}
The script:
- Fetches election data from the API
- Selects the most recent election (or use
--election "{year}"to filter by year) - Downloads all proces-verbaal PDFs
- Generates SHA256SUMS
- No URLs found: Check the REGEX pattern, try a simpler pattern like
\.pdf - 403 Forbidden: Some sites block automated requests; try a different User-Agent or check for an API endpoint
- Missing PREFIX: Check if links are relative or absolute in the HTML source
- Mijn Stembureau API error: Check if API structure changed, update script if needed
Ad hoc scripts are OK for debugging, but if they prove necessary for a municipality or recur across sites:
- Prefer updating the main scripts (e.g.,
scripts/fetch-pv.py) to handle the pattern generally. - If a one-off workaround remains, document the exact steps and rationale in README.md (at the election level or the municipality folder) so future runs are reproducible.
To minimize permission prompts in VS Code, use these consistent command patterns. Always run commands from the repo root and use only relative paths (both in the command and arguments). Do not use absolute paths.
Fetching pages: Do NOT use curl ... | grep pipelines - they cannot be auto-approved. Instead, download to a temp file and inspect separately. Use friendly filenames (letters/numbers/underscores; avoid hyphens or leading dashes) to prevent sed safety blocks. Use a Safari UA to avoid WAF blocks:
mkdir -p tmp
ua='Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15'
curl -sL -A "$ua" "https://www.gemeente.nl/verkiezingen" -o tmp/gemeente.html
grep -i "pdf" tmp/gemeente.htmlThis also avoids repeated downloads when trying different grep patterns.
Quote-safe grep: When searching for PDFs, avoid unescaped quotes in the pattern. Prefer a fixed-string search or wrap patterns with double quotes inside single quotes. If you end up at a dquote> prompt, press Ctrl-C to reset.
rg -i --fixed-strings ".pdf" tmp/gemeente.html
grep -oi 'https:[^"]*\.pdf' tmp/gemeente.html | headRunning fetch scripts (auto-approved):
./scripts/fetch-pv.py {election}/{code}
./scripts/scrapers/mijnstembureau.py https://mijnstembureau-{gemeente}.nl {election}/{code}
./scripts/scrapers/pleio.py <municipality_name> <pleio_url>
./scripts/scrapers/googledrive.py <label>=<url>... {election}/{code}
./scripts/scrapers/stackstorage.py <share_url>... {election}/{code}Never use .venv/bin/python directly; always run the scripts from the repo root to avoid permission prompts.
Some municipalities use Pleio (e.g., hosted by haarlem.pleio.nl) to share election documents. The scripts/scrapers/pleio.py script accesses the Pleio GraphQL API to download PDFs. Pleio folders with public documents are accessible without authentication.
Usage:
./scripts/scrapers/pleio.py <municipality_name> <pleio_url>
# Example:
./scripts/scrapers/pleio.py Zandvoort https://haarlem.pleio.nl/groups/view/.../files/...The folder URL can be found by visiting the municipality's verkiezingen page and looking for a "Pleio" or "file sharing" link.
Some municipalities publish processen-verbaal in public Google Drive folders instead of linking the PDFs directly from their own site. These do not require browser automation or manual ZIP downloads if the folder URLs are known.
Use:
./scripts/scrapers/googledrive.py <label>=<drive_folder_url>... <output_dir>How it works:
- Fetch
https://drive.google.com/embeddedfolderview?id={folder_id}#list - Parse the
file/d/{id}/viewlinks and visible filenames from that HTML - Download each file via
https://drive.google.com/uc?export=download&id={file_id} - Generate a sorted
SHA256SUMS
This worked for Roermond (2026-GR/0957) and reproduced the previously manual
downloads byte-for-byte.
Avoid browser-driven approaches by default. In this repository, static HTML, embedded JSON, public APIs, dedicated fetch scripts, and alternate result pages have consistently been more reproducible than interactive browser tooling.
Only consider browser automation as a last resort after exhausting static/API options, and document the non-browser paths you tried first.
Cleanup and git finalization:
- Prefer editor/file tools for deleting
.lockand.todorather than terminalrmcommands. - Reserve
./scripts/cleanup-municipality.pyfor exceptional rollbacks or bad partial fetches, ideally from the main agent. - For git, stage explicit reviewed files only; avoid whole-directory adds,
git add -f, andgit commit --amendin the municipality-processing flow.
Dutch municipalities have a 4-digit CBS code. Find them at: https://nl.wikipedia.org/wiki/Lijst_van_Nederlandse_gemeenten
The three Caribbean "bijzondere gemeenten" participate in Tweede Kamer elections only (not EU or municipal/GR elections):
| Folder | Name | Website |
|---|---|---|
BES-Bonaire |
Bonaire | https://bonairestemt.nl/ |
BES-Saba |
Saba | https://www.sabagov.nl/ |
BES-SintEustatius |
Sint Eustatius | https://www.statiagovernment.com/ |
Some municipalities have official names that differ from common usage:
| Official Name | Common Name |
|---|---|
| 's-Gravenhage | Den Haag |
| 's-Hertogenbosch | Den Bosch |
Some municipalities include province disambiguation:
- Bergen (Noord-Holland) vs Bergen (Limburg)
We want Proces verbaal documents - the official vote counting forms from each polling station (stembureau). These are typically:
- Named like
{gemeente}_{nummer}_{locatie}_GR26.pdf - One per polling station
- Multi-page PDFs containing detailed vote counts
We also archive:
osv4-3digital counting CSVs when published alongside the PDFs (useful for completeness verification)- Corrigendum/aanpassing documents (corrections) when present
eerste_telling(first count) PDFs when published separately from the final PV
We do NOT want:
- "Uitslagen na controle" (post-verification results) without PV documents
- Standalone result summary pages (HTML)