Skip to content

extend output csv from queries and speedup the queries OData with limit rate +monitoring progress bar - #141

Merged
agrouaze merged 6 commits into
mainfrom
query_extend_csv_output
Jun 23, 2026
Merged

extend output csv from queries and speedup the queries OData with limit rate +monitoring progress bar#141
agrouaze merged 6 commits into
mainfrom
query_extend_csv_output

Conversation

@agrouaze

@agrouaze agrouaze commented Jun 23, 2026

Copy link
Copy Markdown
Member

Description

  • queries are done wrt the quotas CDSE rate limit (2000/min) to speed-up and secure the current solution
  • add progress bar for queries
  • add retry mechanisme for queries
  • end2end test:
match_s1_prodtypes  --input-listing /scale/project/lops-siam-sentinel1-workbench/data/ifremer/listing_products/sentinel1/IW/wind_direction/I
W_SLC_coloc_with_buoys_NDBC_clean.csv --prodtype GRDH --output /home1/scratch/agrouaze/test_match.txt --checkpoint-dir /tmp/                                                                                  
2026-06-23 15:24:46 [INFO] Loaded 2047 product ID(s) from /scale/project/lops-siam-sentinel1-workbench/data/ifremer/listing_products/sentinel1/IW/wind_direction/IW_SLC_coloc_with_buoys_NDBC_clean.csv       
2026-06-23 15:24:46 [INFO] Loaded checkpoint: 920 products already processed                                                                                                                                  
2026-06-23 15:24:46 [INFO] Starting matchup: 1108 product(s) → target type 'GRDH'                                                                                                                             
Matching products: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 1108/1108 [09:49<00:00,  1.88product/s, found=1108, not_found=0, errors=0, rate=39.0] 
2026-06-23 15:34:36 [INFO] Checkpoint saved to /tmp/test_match_checkpoint.txt                                                                                                                                 
2026-06-23 15:34:36 [INFO] Done1108 found, 0 not found, 0 errors. Results/home1/scratch/agrouaze/test_match.txt                                                                                         
2026-06-23 15:34:36 [INFO] Delta-time distribution (secondscount):                                                                                                                                         
2026-06-23 15:34:36 [INFO]    exact : 308                                                                                                                                                                     
2026-06-23 15:34:36 [INFO]       1s : 680                                                                                                                                                                     
2026-06-23 15:34:36 [INFO]       2s : 120                                        

Related Issue

Type of Change

  • 📚 Examples / docs / tutorials / dependencies update
  • 🔧 Bug fix (non-breaking change which fixes an issue)
  • 🥂 Improvement (non-breaking change which improves an existing feature)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 🔐 Security fix

Checklist

  • I've read the CODE_OF_CONDUCT.md document.
  • I've read the CONTRIBUTING.md guide.
  • I've updated the code style using make codestyle.
  • I've written tests for all new methods and classes that I created.
  • I've written the docstring in Google format for all the methods and classes that I used.

@agrouaze agrouaze self-assigned this Jun 23, 2026
@agrouaze agrouaze added the enhancement New feature or request label Jun 23, 2026
@agrouaze
agrouaze requested a review from jean2262 June 23, 2026 16:33
@agrouaze
agrouaze merged commit 2f32a5c into main Jun 23, 2026
7 checks passed
@agrouaze
agrouaze deleted the query_extend_csv_output branch June 23, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

queryCDS should by default output a csv file with S3 path column included.

1 participant