Skip to content

Relocate test data to bucket #994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
visr merged 17 commits into from
Aug 4, 2026
Merged

Relocate test data to bucket #994

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
0a3512b
adding dvc and full update
rleander Jul 10, 2026
dfbcb31
adding dvc and running pixi lock (less changes to lock file than update)
rleander Jul 10, 2026
422e59b
first attempt adding input and output test data to dvc
rleander Jul 10, 2026
16fb743
removing output dir from dvc
rleander Jul 14, 2026
66ad62e
removing aws-cli, not necessary here
rleander Jul 14, 2026
f533135
removed aws-cli and restored the pixi.lock accordingly
rleander Jul 14, 2026
3fdf21b
removing the profile reference from the config file
rleander Jul 20, 2026
8252c83
Merge branch 'master' into relocate_test_data
rleander Jul 24, 2026
7ff8fbe
adding aws credentials for github actions
rleander Jul 24, 2026
e6c8dda
pull dvc replaces download script called from dockerfile
rleander Jul 27, 2026
5fbfa20
run dvc pull from within the pixi environment
rleander Jul 27, 2026
f8561cc
trailing spaces
rleander Jul 27, 2026
93dfa08
update pixi lock and toml, merged master
rleander Jul 28, 2026
4eaafb6
add non-anonymous remote to dvc config for pushing
rleander Jul 31, 2026
a232e79
Merge branch 'main' into relocate_test_data
visr Aug 3, 2026
2865174
Remove Python recommendations
visr Aug 4, 2026
425c71c
Remove AWS secrets from CI
visr Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dvc/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
/config.local
/tmp
/cache
11 changes: 11 additions & 0 deletions .dvc/config
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[core]
remote = wflow-testdata-ro
['remote "wflow-testdata-ro"']
url = s3://wflow/testdata
endpointurl = https://s3.deltares.nl/
allow_anonymous_login = true
region = eu-west-1
['remote "wflow-testdata-rw"']
url = s3://wflow/testdata
endpointurl = https://s3.deltares.nl/
region = eu-west-1
3 changes: 3 additions & 0 deletions .dvcignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Add patterns of files dvc should ignore, which could improve
# the performance. Learn more at
# https://dvc.org/doc/user-guide/dvcignore
2 changes: 2 additions & 0 deletions .github/workflows/CI.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ jobs:
pixi run test-wflow-cov
env:
JULIA_NUM_THREADS: ${{ matrix.threads }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- uses: julia-actions/julia-processcoverage@03114f09f119417c3242a9fb6e0b722676aedf38 # v1.2.2
with:
directories: Wflow/src
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/CIWflowServer.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ jobs:
with:
pixi-version: "latest"
- name: Test WflowServer
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
pixi run test-wflow-server
- uses: julia-actions/julia-processcoverage@03114f09f119417c3242a9fb6e0b722676aedf38 # v1.2.2
Expand Down
1 change: 0 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ LocalPreferences.toml

# Temporary files
/dev/
/Wflow/test/data/
/tmp/
Wflow/test/sbm_piave_config-debug.toml
Wflow/test/sbm_simple_temp.toml
Expand Down
4 changes: 1 addition & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,7 @@ RUN apt-get update && apt-get install -y \
ADD . /app
WORKDIR /app/build/create_binaries/
RUN julia --project=/app/Wflow -e "using Pkg; Pkg.instantiate();" && \
julia --project=/app/Wflow /app/utils/download_test_data.jl && \
julia --project -e "using Pkg; Pkg.instantiate()" && \
julia --project create_app.jl && \
rm -rf /app/test/data/*
julia --project create_app.jl

ENTRYPOINT [ "/app/build/create_binaries/wflow_bundle/bin/wflow_cli" ]
2 changes: 2 additions & 0 deletions Wflow/test/data/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/input
/output
6 changes: 6 additions & 0 deletions Wflow/test/data/input.dvc
View file
Open in desktop

@visr visr Jul 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you intent to bundle the whole input directory together? I think that means that if one file changes, everything gets re-downloaded. Not too bad right now at 300MB, but this could get annoying. We can also do it per file. Or perhaps per model, but then we should probably bundle the models in subdirectories.

@vers-w vers-w Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now we bundle all files (version): https://github.com/visr/wflow-artifacts/releases/tag/v1.0.0.

@visr visr Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think it is fine like this. If it starts to get annoying because people are changing different files in different branches, we can still decide to version them separately.
And it doesn't seem to be true that everything gets redownloaded.

@visr visr Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I see why I was wrong. DVC doesn't track the dir as one blob, it just stores a JSON with pointers to the actual files, see this eac653f3009e1f9babcae88c07ee56.dir for instance:

[
   {
      "md5":"76afca075bef6e20f48eb11dd585d7a4",
      "relpath":"hydrotype.dbf"
   },
   {
      "md5":"cb3e648c38fd75188df5ea80c1924386",
      "relpath":"hydrotype.prj"
   },
   {
      "md5":"b5ed91a25adf2925fc573061e948d6ff",
      "relpath":"hydrotype.shp"
   },
   {
      "md5":"2806f374363203bf1e88ce3b3ee238f2",
      "relpath":"hydrotype.shx"
   },
   {
      "md5":"399c079845032dfa1749335c619725e3",
      "relpath":"vdGaast_water_depth.csv"
   }
]

Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
outs:
- md5: 88ca646c32a3859e619362a55fe8bdca.dir
size: 317173497
nfiles: 19
hash: md5
path: input
3,622 changes: 3,600 additions & 22 deletions pixi.lock
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion pixi.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ install-prek = "prek install --overwrite"
update-manifest-julia = "julia --project utils/update-manifest.jl"
install = { depends-on = ["install-prek"] }
# Test data
download-test-data = { cmd = "julia --project ../utils/download_test_data.jl", cwd = "Wflow" }
download-test-data = { cmd = "dvc pull", cwd = "." }
# Build
build-wflow-cli = { cmd = "julia --project create_app.jl", cwd = "build/create_binaries", depends-on = [
{ "task" = "instantiate-julia", "args" = [
Expand Down Expand Up @@ -72,6 +72,8 @@ inputs = ["Manifest.toml"]
outputs = ["Wflow.spdx.json"]

[dependencies]
dvc = "*"
dvc-s3 = "*"
julia = { version = "==1.12.6", channel = "https://prefix.dev/julia-forge" }
python = ">=3.10"
jupyter = ">=1.1.1,<2"
Expand Down
63 changes: 0 additions & 63 deletions utils/download_test_data.jl
View file
Open in desktop

This file was deleted.

Loading