[FEATURE] Conditional HTTP GET for Blacklist Downloads (ETag / Last-Modified) #3282

[mkf-sysangels]

[FEATURE] Conditional HTTP GET for Blacklist Downloads (ETag / Last-Modified) #3282

[Copilot]

Pull request overview

This PR implements conditional HTTP GET support (via ETag / Last-Modified headers) for the blacklist download job in BunkerWeb. Instead of performing a full download on every job run after the 1-hour TTL expires, the job now sends If-None-Match or If-Modified-Since headers to allow servers to respond with a lightweight 304 Not Modified. A HEAD-request-based Content-Length comparison is included as a last-resort fallback when neither ETag nor Last-Modified are available. The feature reduces bandwidth, CPU load, and load on upstream list providers.

Changes:

• New parse_http_metadata() helper to extract ETag, Last-Modified, and Content-Length from cached file headers.
• New build_cache_header() helper to persist HTTP metadata (ETag, Last-Modified, Content-Length) in the cache file's comment header block.
• Updated download loop to send conditional GET headers, handle 304 responses by refreshing the cache timestamp, and fall back to a HEAD request + Content-Length comparison when no standard validators are present. Also upgrades several log calls from warning to error where appropriate.

[mkf-sysangels]

Hello @TheophileDiot

the PR is ready for code audit and public preview

[TheophileDiot]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

Warning

Rate limit exceeded

@TheophileDiot has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 7 minutes and 1 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 7 minutes and 1 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6191c94d-da5c-4a1b-9725-ed3f2357f715

📥 Commits

Reviewing files that changed from the base of the PR and between 6d840d7 and c7c1718.

📒 Files selected for processing (1)

• src/common/core/blacklist/jobs/blacklist-download.py

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[TheophileDiot]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.