Skip to content

Keep integers on the python read path for fixed-shape ArrayXD columns with nulls#8363

Open
ebarkhordar wants to merge 1 commit into
huggingface:mainfrom
ebarkhordar:fix/arrayxd-fixeddim-to-numpy-int-float-cast
Open

Keep integers on the python read path for fixed-shape ArrayXD columns with nulls#8363
ebarkhordar wants to merge 1 commit into
huggingface:mainfrom
ebarkhordar:fix/arrayxd-fixeddim-to-numpy-int-float-cast

Conversation

@ebarkhordar

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #8362.

A fixed-shape ArrayXD column (Array2D(shape=(1, 2), dtype="int64"), and so on) that contains a None row is silently cast to float on the python read path: to_dict() / to_pylist() return every value as a float, and integers above 2**53 are altered, because an unrelated row is null. dataset[i][column] returns the correct integers, so the result depends on whether another row happens to be null.

Root cause

ArrayExtensionArray.to_numpy casts a fixed-shape column to float64 so it can insert np.nan for the null rows (a dense numpy array cannot hold both integers and a missing marker). to_pylist returns numpy_arr.tolist(), so that float cast reaches the python read path, where None is available and no cast is needed.

Fix

In to_pylist, when the array is fixed-shape and has nulls, build the list from the nested-list storage (self.storage.to_pylist()), which already carries the exact values and None for the null rows.

The invariant: a non-null value read through the python path must never be altered because another row in the column is null.

Scope kept deliberately narrow:

  • The numpy format path is unchanged. to_numpy still returns the dense float64 + np.nan array (asserted by the existing test_array_xd_with_none), so with_format("numpy") behaves exactly as before.
  • The dynamic-first-dim branch (shape=(None, ...)) is untouched.

Verification

  • New test test_array_xd_with_none_python_format_keeps_integers: fails on main ([[nan, nan]] != None), passes here. It covers integer preservation, the 2**53 + 1 precision case, and asserts the numpy format still returns a dense float64 + nan array.
  • python -m pytest tests/features/test_array_xd.py: 103 passed.
  • ruff check and ruff format --check clean on the changed files.
  • Verified in a clean python:3.11-slim container at HEAD 521a590. Not verified: the torch/tf/jax formatting paths, which this change does not touch (it only affects the python to_pylist path).

… with nulls

A fixed-shape ArrayXD column that contains a null row was silently cast to
float on the python read path (to_dict / to_pylist): non-null integers came
back as floats and values above 2**53 lost precision, because an unrelated
row was null. Build the python list from the nested-list storage in that
case, which keeps the exact values and returns None for the null rows.

The numpy format path is unchanged (to_numpy still returns the dense float64
array with np.nan for nulls), and the dynamic-first-dim branch is untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ArrayXD fixed-shape integer column with a null row is silently cast to float on the python read path

1 participant