Investigate DuckDB TOP_N + Parquet behavior behind large-sort I/O

[Copilot]

Context

Large ORDER BY ... LIMIT queries on 10M–20M parquet rows show multi-GB transfer and minute-scale latency despite DuckDB planning TOP_N. This PR captures root-cause evidence from DuckDB source to explain why row-group pruning is weaker than expected for this query shape.

Changes

• Planner/engine behavior analysis
  • Confirmed LIMIT + ORDER BY is rewritten to LOGICAL_TOP_N and pushes a scan-side dynamic filter.
  • Confirmed dynamic filter starts uninitialized and only tightens after boundary discovery in PhysicalTopN.
• Parquet pruning path analysis
  • Verified row-group skip occurs only when stats pruning returns FILTER_ALWAYS_FALSE; otherwise row groups are read.
  • Verified dynamic filter is wrapped in optional filter semantics, reducing early hard-prune behavior.
• I/O behavior correlation
  • Linked high transfer volume to incremental boundary tightening + broad projection (SELECT *) + optional-filter handling in parquet scan/prefetch paths.
• Actionable observability
  • Identified SkipRowGroup vs ReadRowGroup operator logging path as the direct way to see exactly which row groups are scanned.

SELECT *
FROM 'mytable.parquet'
ORDER BY "File Size" DESC, "file_row_number" ASC
LIMIT 52;

Testing

Not applicable — no repository code changes were introduced; this PR is an investigation/evidence update.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• production.int.allencell.org
  • Triggering command: /usr/local/bin/node node /home/REDACTED/work/biofile-finder/biofile-finder/node_modules/.bin/mocha --exit (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)