MultiEpochsDataloader

[Filos1992]

Hey everyone,
I've reused the MultiEpochsDataLoader because the normal Dataloader slows down after every epoch and it works just as desired.
Is there any reason why this isn't a standard Pytorch class? are there any Problems with type of dataloading?
Best greetings,
Filos

[AhmedAhmedEG]

2 years and that question never got answered, I can't imagine how much pressure PyTorch devs have to deny requests for such a looong period.

[AhmedAhmedEG]

And yes, MultiEpochsDataLoader is amazing, and it shrinked a period of three days for my model to finish training, to 6 hours, by using 8 workers only, Imagine if I added more, It's very weird why such and amazing and polished library like PyTorch turned their gaze over such a critical problem, I can't imagine they was respawning all workers and reinitializing datasets in them for every iteration, this is unbelievable how they left it like that.

[antoinebrl]

Pytorch Dataloader has a persistent_workers argument since 1.7.0. https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

[AhmedAhmedEG]

Damn, I didn't seen it anywhere until I looked at the docs, well, the fact those questions never got answered led me to wrongly think it has never fixed.