NASDAQ-100 minute bars data preperation (inside the docker container) not working

[martinfrei-web]

Installing NASDAQ-100 minute bars doesn't seem to work as /data/equities/market/nasdaq100/minute_bars folder gets not created. Following all the output I get:

(ml4t) ubuntu@6f9caded0fc0:/app$ uv run python data/equities/market/algoseek_convert.py     --dataset nasdaq100-minute-bars --source data/nasdaq-100-constituents-taq-ext.zip
warning: `VIRTUAL_ENV=/Users/martin/dev/workspace/machine-learning-for-trading/.venv` does not match the project environment path `/opt/ml4t` and will be ignored; use `--active` to target the active environment instead
Using data path (ML4T_DATA_PATH): /data

============================================================
ALGOSEEK CONVERSION - nasdaq100-minute-bars
============================================================
[06:15:41] Source: data/nasdaq-100-constituents-taq-ext.zip
[06:15:41] Output: /data
[06:15:41] 505 days across 24 months -> /data/equities/market/nasdaq100/minute_bars

[martinfrei-web]

And I miss an empty entry for QUANDL_API_KEY= inside .env.example. Isn't that required anymore?

[stefan-jansen]

Thanks for the report - both points are real, and the first one is not what it looks like.

The conversion was running. Nothing is written until a whole month is concatenated, and that
write is what creates minute_bars/, so during the longest phase of the run there is no output and
no directory to check. Only a failed day logged anything; a successful one was silent. With 505
days across 24 months, the first month alone is ~21 days of ~504 symbol files, which is a long time
to sit at that last line with nothing to look at.

If the process was still alive, it was making progress. du -sh /data/equities/market/nasdaq100
after the first month lands is the quickest way to confirm that on the current release.

Fixed in 05c95d5: the converter now logs each day as it completes with its row count and its
position in the month, and announces the concatenate-and-sort step before it starts, so the quiet
phase is visible while it happens. The same commit drops each month's per-day frames once they are
concatenated - a month of minute bars for the full constituent list is the largest object the script
builds, so holding both copies at once was worth removing.

On QUANDL_API_KEY: you are right, and the template was the thing that was wrong. It is still
required - download.py tells you to add it and the loader declares it - so its absence from
.env.example was a straight omission. Added in the same commit, with a note that
NASDAQ_DATA_LINK_API_KEY is read as an alias.

One thing that is not a problem, since it appears in your output: the VIRTUAL_ENV warning means
uv ignored your host's .venv path and used the container's /opt/ml4t, which is what you want
inside Docker.

Both changes are on a branch and will reach main with the next merge.