fix(conflict): restore keyless GDELT fallback#5269
Conversation
Strix Security ReviewNo security issues found. Updated for Reviewed by Strix |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR restores a keyless GDELT conflict signal by adding a new
Confidence Score: 4/5The change is fail-closed: if both DOC and bulk paths fail the existing last-good/retry contract is preserved, timing math is regression-pinned, and the invariant holds. The bulk fallback module is solidly implemented with URL allowlisting, per-byte size caps, MD5 integrity, ZIP flag checks, and decompression output limits. The pagination object reports The bulk fallback integration in Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Seeder as seed-conflict-intel
participant DOC as GDELT DOC 2.0
participant Bulk as _conflict-gdelt-bulk
participant GCS as GCS storage.googleapis.com
Seeder->>DOC: fetchGdeltCountryEvents x20 countries batched
alt DOC sweep OK AND events not empty
DOC-->>Seeder: events array
Seeder->>Seeder: return events source undefined
else DOC below floor OR zero events
DOC-->>Seeder: throttled or empty
Seeder->>Bulk: fetchBulkEvents
Bulk->>GCS: "GET masterfilelist.txt Range bytes=-16384"
GCS-->>Bulk: 206 manifest tail
Bulk->>Bulk: parseGdeltRecentExports 8 descriptors
loop 8 exports concurrency 4
Bulk->>GCS: GET export.CSV.zip
GCS-->>Bulk: ZIP bytes
Bulk->>Bulk: verify size MD5
Bulk->>Bulk: extractGdeltExportCsv inflateRawSync
Bulk->>Bulk: mapGdeltExportToConflictEvents
end
alt at least one export succeeded
Bulk-->>Seeder: events exportTimestamp
Seeder->>Seeder: return source gdelt-bulk
else all exports failed
Bulk-->>Seeder: throw all exports failed
Seeder->>Seeder: throw combined error
end
end
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Seeder as seed-conflict-intel
participant DOC as GDELT DOC 2.0
participant Bulk as _conflict-gdelt-bulk
participant GCS as GCS storage.googleapis.com
Seeder->>DOC: fetchGdeltCountryEvents x20 countries batched
alt DOC sweep OK AND events not empty
DOC-->>Seeder: events array
Seeder->>Seeder: return events source undefined
else DOC below floor OR zero events
DOC-->>Seeder: throttled or empty
Seeder->>Bulk: fetchBulkEvents
Bulk->>GCS: "GET masterfilelist.txt Range bytes=-16384"
GCS-->>Bulk: 206 manifest tail
Bulk->>Bulk: parseGdeltRecentExports 8 descriptors
loop 8 exports concurrency 4
Bulk->>GCS: GET export.CSV.zip
GCS-->>Bulk: ZIP bytes
Bulk->>Bulk: verify size MD5
Bulk->>Bulk: extractGdeltExportCsv inflateRawSync
Bulk->>Bulk: mapGdeltExportToConflictEvents
end
alt at least one export succeeded
Bulk-->>Seeder: events exportTimestamp
Seeder->>Seeder: return source gdelt-bulk
else all exports failed
Bulk-->>Seeder: throw all exports failed
Seeder->>Seeder: throw combined error
end
end
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Fixed in |
1 participant
Summary
Without ACLED credentials, conflict seeding no longer collapses when GDELT DOC queries are throttled or return a misleading all-empty result. The seeder now falls back to GDELT's recent 15-minute event exports and produces the same priority-country conflict shape consumed by the escalation EMA.
The fallback range-reads the current manifest, downloads only the eight newest exports through the TLS storage origin, and bounds every network and decompression step. Manifest URLs, byte sizes, checksums, ZIP metadata, GDELT event classes, and priority-country mappings are validated before publication; if both DOC and bulk paths fail, the existing last-good preservation/retry contract remains fail-closed. The seeder lock was also resized and regression-pinned to cover the bulk leg after the parallel auxiliary feeds settle.
Related: #5256
Validation
20260713113000, 8/8 exports succeeded, 94 material-conflict records across 9 priority countries.npm run build:fullcompleted successfully.Post-deploy monitoring
GDELT bulk conflict-events fallbackwith a non-zero record count.acled-intelseed metadata stays current and non-zero for two 30-minute intervals.bulk fallback failedorRETRY FAILED; if repeated, inspect storage-origin reachability and manifest/schema drift before rerunning the seeder.