Skip to content

Decode Json() columns in Dataset.to_pandas()#8344

Open
ebarkhordar wants to merge 1 commit into
huggingface:mainfrom
ebarkhordar:fix/to-pandas-decode-json-columns
Open

Decode Json() columns in Dataset.to_pandas()#8344
ebarkhordar wants to merge 1 commit into
huggingface:mainfrom
ebarkhordar:fix/to-pandas-decode-json-columns

Conversation

@ebarkhordar

Copy link
Copy Markdown
Contributor

What does this PR do?

Dataset.to_pandas() returned the raw Arrow storage strings for Json() feature columns, while every other in-memory export path decodes them. to_dict(), to_list(), to_json() and with_format("pandas") all apply get_json_field_paths_from_feature + json_decode_field; to_pandas() skipped that step, so it alone produced '{"a":1}' (str) where the siblings produce {'a': 1} (dict). Nested List(Json()) columns had the same gap.

This applies the same decode after the Arrow-to-pandas conversion, in both the single and batched code paths, so a Json() column materializes as decoded Python objects (pandas object dtype), consistent with the other export methods.

Before:

ds = Dataset.from_dict({"col": [{"a": 1}]}, features=Features({"col": Json()}))
ds.to_dict()["col"][0]      # {'a': 1}
ds.to_pandas()["col"][0]    # '{"a":1}'  (str)  <- fixed to {'a': 1}

Fixes #8343

How was it tested?

  • Added test_to_pandas_decode_json and test_to_pandas_decode_nested_json next to the existing to_dict/to_list decode tests, covering a top-level Json() column (including a None value), a nested object, List(Json()), and the batched generator path. They fail on main (to_pandas returns raw strings) and pass with this change.
  • Verified in a clean container on this commit: the two new tests fail before the fix and pass after; the existing to_dict/to_list decode, json_feature, to_pandas and to_polars tests still pass.
  • ruff check and ruff format --check are clean on the changed files.

Not covered (possible follow-up)

to_polars() shows the same raw-JSON-string behavior, but I left it out of this PR deliberately. Unlike pandas, whose object dtype holds arbitrary decoded objects faithfully, polars has no per-cell Python-object column, and letting it infer a schema is lossy for heterogeneous Json() data: [{"a": 1}, {"b": [2, 3]}] collapses to Struct({'a': Int64}) and the second row becomes {'a': None}, silently dropping b. The faithful alternative is a pl.Object column, which is a representation choice I would rather leave to you. Happy to send a separate PR for to_polars() if you confirm the behavior you want.

to_pandas() returned raw Arrow storage strings for Json() columns, while
to_dict(), to_list(), to_json() and with_format("pandas") all decode them via
get_json_field_paths_from_feature + json_decode_field. Apply the same decode
after the Arrow->pandas conversion, in both the single and batched paths, so a
Json() column materializes as decoded Python objects. Also covers nested
List(Json()).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset.to_pandas() returns raw JSON strings for Json() columns instead of decoded objects

1 participant