Decode Json() columns in Dataset.to_pandas()#8344
Open
ebarkhordar wants to merge 1 commit into
Open
Conversation
to_pandas() returned raw Arrow storage strings for Json() columns, while
to_dict(), to_list(), to_json() and with_format("pandas") all decode them via
get_json_field_paths_from_feature + json_decode_field. Apply the same decode
after the Arrow->pandas conversion, in both the single and batched paths, so a
Json() column materializes as decoded Python objects. Also covers nested
List(Json()).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Dataset.to_pandas()returned the raw Arrow storage strings forJson()feature columns, while every other in-memory export path decodes them.to_dict(),to_list(),to_json()andwith_format("pandas")all applyget_json_field_paths_from_feature+json_decode_field;to_pandas()skipped that step, so it alone produced'{"a":1}'(str) where the siblings produce{'a': 1}(dict). NestedList(Json())columns had the same gap.This applies the same decode after the Arrow-to-pandas conversion, in both the single and batched code paths, so a
Json()column materializes as decoded Python objects (pandasobjectdtype), consistent with the other export methods.Before:
Fixes #8343
How was it tested?
test_to_pandas_decode_jsonandtest_to_pandas_decode_nested_jsonnext to the existingto_dict/to_listdecode tests, covering a top-levelJson()column (including aNonevalue), a nested object,List(Json()), and the batched generator path. They fail onmain(to_pandasreturns raw strings) and pass with this change.to_dict/to_listdecode,json_feature,to_pandasandto_polarstests still pass.ruff checkandruff format --checkare clean on the changed files.Not covered (possible follow-up)
to_polars()shows the same raw-JSON-string behavior, but I left it out of this PR deliberately. Unlike pandas, whoseobjectdtype holds arbitrary decoded objects faithfully, polars has no per-cell Python-object column, and letting it infer a schema is lossy for heterogeneousJson()data:[{"a": 1}, {"b": [2, 3]}]collapses toStruct({'a': Int64})and the second row becomes{'a': None}, silently droppingb. The faithful alternative is apl.Objectcolumn, which is a representation choice I would rather leave to you. Happy to send a separate PR forto_polars()if you confirm the behavior you want.