Skip to content

MetadataConfigs drops parquet shards when exported config rows are non-consecutive #8269

Description

@KirtiRamchandani

Bug

_from_exported_parquet_files_and_dataset_infos() uses itertools.groupby() on unsorted exported_parquet_files.

If the same config appears again after another config, the earlier shard URLs are lost.

Repro

from itertools import groupby
from operator import itemgetter

exported = [
    {"config": "default", "split": "train", "url": ".../train-0000.parquet"},
    {"config": "other", "split": "train", "url": ".../other/train.parquet"},
    {"config": "default", "split": "train", "url": ".../train-0001.parquet"},
]

metadata_configs = {
    config_name: [p["url"] for _, ps in groupby(rows, itemgetter("split")) for p in ps]
    for config_name, rows in groupby(exported, itemgetter("config"))
}
# default keeps only train-0001

Proof (failing before fix)

FAILED - default only had 1 URL instead of 2

Fix

Sort by (config, split) before both groupby() calls.

I'll follow up with a PR + regression test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions