You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Browse filesBrowse the repository at this point in the historyBrowse files
authored
Add alternative source for ERA5 data to eliminate queueing (#507)
* initial attemtp at opendap access
* smarter CE plus semaphore to limit no of simultaneous connections
* add caching
* add deps for ncar
* zarr instead of netcdf for cahing to improve parallelism, tests to compare against existing era5 cutout
* test setup simplified, minor fix to downloaded coords
* first pass at spatial subsetting
* processing for solar
* first attempt at regrid
* regridding fixed
* processing for remaining values implemented, passes tests at native resolution
* merged tests for era5_ncar
* precomputed time axis to save on fetches, parallel building, skip interpolation if grids are aligned, grid interpolation sanitise only if interpolatingfor consistency with cds
* rate limit downloads through a custom dask pool instead of semaphore
* todos after code review
* style updated for consistency with era5.py, type annotations
* added units to missing vars
* invariant import handling simplified@
* per-variable tolerances for weird resolution cutout
* numpy style docstrings
* add time sampling tests
* first pass at era5_ncar doc
* finished docs
* suppress pydap log, add regridding warning
* updated comments + reorganised function - update incomplete
* use circular azimuth error in tests, better zero meridian handling, comments
* fix documentation typos
* clearer comment structure, fix for edge case when retrieving data around 0 longitude index
* add release note
* move release note to correct place
* clean up comments in era5_ncar tests
* consistent formatting of log output in era5_ncar, suppress known warnings
* clean up doc notebook output
* more efficient zarr cache chunking, minor refactoring of logging setup
* add all atlite vars downloaded from edh plus tests
* update docs for EDH model
* customizable chunk size support, cast all vars to float32 to save on space
* udpate test tols for compatibility with float32 cast
* updated documentation in notebooks
* clearer comments
* retries on failed chunk downloads, minor refactors
* docs improvements
* fix typo in test docstring
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* replace fsspecstore with obstore for more robust connection
* add EDH key to test CI
* bump python version to support zarr v3
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update classifiers in pyproject to remove 3.10
* remove unnecessary tempdir from test on edh
* performance optimisation to reduce ram usage. more sensible chunk size, reverse subset order, better retry config
* rechunk for memory use optimisation, add example on using distributed scheduler to download large cutouts
* fix example doc to render properly with pandoc
* fix CI runners getting bounced from EDH due to multiple machines attempting to reach it at the same time with same creds
* fix type errors in era5-edh
* fix types and docstrings to match new ruff rules
* move era5-edh to an upcoming release in release notes
* refactor: reduce test retry timeout to fail faster in case of edh server/credential issue
* chore: remove typing_extensions dependency post upgrade to python 3.11
* docs: fix typos in example and installation instructions
Co-authored-by: Florian Maurer <fmaurer@disroot.org>
* refactor: restore long edh reach retry times to deal with EDH timing out in CI, add a warning
in the CI test job 9x runners simultaneously hammer EDH with the same
API keys, which results in 403s. for now dealt with via long timeouts,
final fix would be a two-step test process, separate creds, or some
other form of rate limiting
* refactor(era5-edh): share wind/influx processing with era5, add offline tests
- Extract _process_wind/_process_influx helpers in era5.py (single_precision flag)
- Add offline tests for resolution guard and credential resolution
- Parametrize era5-edh comparison tests; fix _edh_reachable auth handling
- Remove redundant comments; document ignored CDS-only get_data params
* docs: add era5-edh usage note to release notes
* refactor: edh reachability test retries on 429s
sometimes many CI runners hit EDH at the same time, which results in
429s and rate limiting. retries are useful in that case
---------
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Florian Maurer <fmaurer@disroot.org>
Co-authored-by: Fabian <fab.hof@gmx.de>
0 commit comments