fix: disable persistent_workers when num_workers is 0 (mllam#235)

Co-authored-by: Simon Adamov <kode@mailbox.org>

Author: santhil-cyber

CHANGELOG.md

@@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Fix README image paths to use absolute GitHub URLs so images display correctly on PyPI [\#188](https://github.com/mllam/neural-lam/pull/188) @bk-simon
 
+- Changed the hardcoded True to a conditional check "persistent_workers=self.num_workers > 0" [\#235](https://github.com/mllam/neural-lam/pull/235) @santhil-cyber
+
 ### Maintenance
 
 - Update PR template to clarify milestone/roadmap requirement and maintenance changes [\#186](https://github.com/mllam/neural-lam/pull/186) @joeloskarsson

neural_lam/weather_dataset.py

@@ -672,7 +672,7 @@ def train_dataloader(self):
             num_workers=self.num_workers,
             shuffle=True,
             multiprocessing_context=self.multiprocessing_context,
-            persistent_workers=True,
+            persistent_workers=self.num_workers > 0,
         )
 
     def val_dataloader(self):
@@ -683,7 +683,7 @@ def val_dataloader(self):
             num_workers=self.num_workers,
             shuffle=False,
             multiprocessing_context=self.multiprocessing_context,
-            persistent_workers=True,
+            persistent_workers=self.num_workers > 0,
         )
 
     def test_dataloader(self):
@@ -694,5 +694,5 @@ def test_dataloader(self):
             num_workers=self.num_workers,
             shuffle=False,
             multiprocessing_context=self.multiprocessing_context,
-            persistent_workers=True,
+            persistent_workers=self.num_workers > 0,
         )