Is NaFlex IterableDataset support available to contribute?

[TomasGuija]

Hey there,

I’ve been looking for opportunities to contribute to the Hugging Face ecosystem and recently started exploring timm’s NaFlex implementation.

While reading through the data pipeline, I noticed that iterable-dataset support was listed as unfinished in #2466. Current naflex_dataset.py also still contains the NaFlexIterableDatasetWrapper TODO.

This seems like an interesting and useful problem to work on. I’d be interested in taking it on as a contribution. Before investigating the implementation in greater depth, I wanted to check whether anyone is already working on this, publicly or internally, and whether an external contribution would still be welcome.

I have some initial ideas, but I would prefer to confirm that the work is available and all before going further.

Thanks!

[rwightman]

@TomasGuija I added support to OpenCLIP to for NaFlex w/ webdataset for image + text / audio + text datasets. The timm support would be likely look similar to aspects of the batcher / scheduler there... it's non-trival in any case, but if it's something you need it's worth looking into. Nobody else has asked that I'm aware of...