API/BUG: support for operations with custom objects / object dtype

[jorisvandenbossche]

Consider the following dummy example of a custom Python object that supports some arithmetic operations:

class MyObject:
    def __init__(self, val):
        self.val = val
    def __add__(self, other):
        if hasattr(other, "dtype"):
            return NotImplemented
        return MyObject(self.val + other)
    def __radd__(self, other):
        if hasattr(other, "dtype"):
            return NotImplemented
        return MyObject(other + self.val)
    def __repr__(self):
        return f"<MyObject({self.val})>"

Working with such objects in pandas containers generally works, i.e. we defer to the scalar operation and assemble the results:

# operation of scalar/arr/series with numeric other
arr = np.array([MyObject(1), MyObject(2)])
ser = pd.Series([1, 2])

print(ser + arr[0])
print(ser + arr)
print(ser + pd.Series(arr))
print(arr[0] + ser)
print(arr + ser)
print(pd.Series(arr) + ser)

When the other operand were strings, this worked in pandas 2.x as well:

# operation of scalar/arr/series with string (object) other
arr = np.array([MyObject("a"), MyObject("b")])
ser = pd.Series(["1", "2"])

print(ser + arr[0])
print(ser + arr)
print(ser + pd.Series(arr))
print(arr[0] + ser)
print(arr + ser)
print(pd.Series(arr) + ser)

However, this case no longer works in pandas 3.0 with the default str dtype.

We specifically fixed this for pathlib.Path objects for the scalar case (#61940 / #62229), but then got a report for such objects in a Series (#63832). But in the end, while we can "fix" this again specifically for Path objects, this is a more general issue with any generic Python object.

I think in general pandas has always been quite flexible in supporting custom objects, and IMO we should continue to do that (yes, you can define ExtensionDtypes for full control over handling of custom objects, but that is often overkill). Once we detect something we cannot infer to a non-object dtype, we can use a slower element-wise code path. And IMO we should keep doing that also for newer dtypes (such as now str dtype).

[sanrishi]

Hi @jorisvandenbossche I see this is a regression for 3.0. Are you planning to handle this yourself, or would you like me to investigate a fix?

[jorisvandenbossche]

Discussed this on the community meeting briefly, and there was a general agreement that we should continue to support this use case (@rhshadrach @mroeschke @Dr-Irv @jbrockmendel)

While starting to look at it, one aspect we have to decide on: do we want to have missing values propagate automatically, or have them operate with the custom object? (which might typically error)

For example, using the pathlib objects for which we added custom support. For string dtype, @jbrockmendel added support for auto-propagating missing values:

>>> path / pd.Series(["a", None, "b"])
0    /home/a
1        NaN
2    /home/b
dtype: object

but with object dtype, it calls the element path / None which raises an error:

>>> path / pd.Series(["a", None, "b"], dtype=object)
...
TypeError: unsupported operand type(s) for /: 'PosixPath' and 'NoneType'

[rhshadrach]

I'm good with carving out propagation for Python stdlib classes, but think 3rd party classes should receive the value as is. They can implement operators for working with them if they want it to function.

[Dr-Irv]

I'm good with carving out propagation for Python stdlib classes, but think 3rd party classes should receive the value as is. They can implement operators for working with them if they want it to function.

I agree

[jorisvandenbossche]

While I understand such a distinction (you can't modify stdlib objects to handle NA or NaN), I am not sure that distinguishing this in practice will be feasible.

First, for series / scalar ops, we can of course check the scalar (assuming we devise some way to check if an object's module is in the stdlib), so that is relatively doable.
But for the the series / series case, we would have to kind of infer if the series contains stdlib objects or not, to know which code path to take? (like, we would have to add support to lib.infer_dtype for stdlib objects, similarly as it already infers Decimals)

[jorisvandenbossche]

Actually, if both left and right are object dtype, we already do use a masked operation (i.e. propagating missing values automatically without operating on them). But so that currently only happens if the left or right operand is object dtype, and only happens if it first failed without the masking.

pandas/pandas/core/ops/array_ops.py

Lines 219 to 231 in 4728afb

	try:
	result = func(left, right)
	except TypeError:
	if not is_cmp and (
	left.dtype == object or getattr(right, "dtype", None) == object
	):
	# For object dtype, fallback to a masked operation (only operating
	# on the non-missing values)
	# Don't do this for comparisons, as that will handle complex numbers
	# incorrectly, see GH#32047
	result = _masked_arith_op(left, right, op)
	else:
	raise

So for example with the custom object from the top post:

# numeric right -> never tries masked because doing the addition doesn't fail (`2 + np.nan = np.nan`)
>>> pd.Series([MyObject(1), MyObject(2)], dtype=object) + pd.Series([1, np.nan])
0    <MyObject(2.0)>
1    <MyObject(nan)>
dtype: object
# object right -> never tries masked because doing the addition doesn't fail (`2 + np.nan = np.nan`)
>>> pd.Series([MyObject(1), MyObject(2)], dtype=object) + pd.Series([1, np.nan], dtype=object)
0      <MyObject(2)>
1    <MyObject(nan)>
dtype: object
# object right with None -> here the addition does fail (`2 + None`), so it retries with a masked operation
>>> pd.Series([MyObject(1), MyObject(2)], dtype=object) + pd.Series([1, None], dtype=object)
0    <MyObject(2)>
1              NaN
dtype: object

Or for Path objects:

# here using None or np.nan doesn't matter, as both give an error -> retry with masked operation
>>> pd.Series([Path("/home/"), Path("/home/")], dtype=object) / pd.Series([Path("user1"), np.nan], dtype=object)
0    /home/user1
1            NaN
dtype: object
>>> pd.Series([Path("/home/"), Path("/home/")], dtype=object) / pd.Series([Path("user1"), None], dtype=object)
0    /home/user1
1            NaN
dtype: object

Some thoughts on this current behaviour: (EDIT: those were wrong, so deleted)
The series / scalar case for a Path object fails right now, but that seems more like a bug in _masked_arith_op (it actually does retry with the masked version)

[rhshadrach]

like, we would have to add support to lib.infer_dtype for stdlib objects

I would create a new function for this. Very loosely, something like

def make_stdlib_obj_mask(arr):
    assert arr.dtype == "object":
    mask = np.empty_like(arr, dtype="bool")
    for i, obj in enumerate(arr):
        mask[i] = isinstance(obj, (Path, ...))
    return mask

When I said carving out an exception, I meant on an as-needed basis. I expect very few stdlib objects will need this.

Alternatively, we could add a dunder that users can put in their custom objects.

class Foo:
    __pandas_supports_na_ops__ = True

and when this is present, we will pass NA values in ops otherwise we mask them.

[jorisvandenbossche]

Given that we essentially already do masking (just with retry) for any scalar object that passes the lib.is_scalar check (which already includes a lot of stdlib objects, like all datetime, decimal, fractions, etc objects), we should probably at least keep doing that for those objects, also when one of the operands is str dtype (the original issue that triggered me to open this issue).

Short term, an easy "solution" for the specific pathlib case could be to update is_scalar to recognize pathlib.Path objects, and then to update the StringArray arithmetic ops to match how it currently works with object dtype.

That doesn't yet solve things for all custom scalar objects in general, but at least fixes it for Path objects with a generic solution (i.e. something that works for all scalars recognized by is_scalar, without the current "special case" code paths for just pathlib.Path)

[Dr-Irv]

That doesn't yet solve things for all custom scalar objects in general, but at least fixes it for Path objects with a generic solution (i.e. something that works for all scalars recognized by is_scalar, without the current "special case" code paths for just pathlib.Path)

Another idea would be to inspect the type of the object and see if the dunder method is defined. E.g., for example with Path, only __truediv__ is supported, not __add__ or __sub__, etc.

[jbrockmendel]

Actually, if both left and right are object dtype, we already do use a masked operation

I think this is a consequence of not having nullable ints/bools and can be deprecated after we are nullable-by-default.

In the specific case of Series[object].__op__(Series[object]) i don't think we should be masking, just operate pointwise.

[sanrishi]

Another idea would be to inspect the type of the object and see if the dunder method is defined. E.g., for example with Path, only truediv is supported, not add or sub, etc.

One challenge with introspection: even when the dunder exists the operation can still fail with None/NA values. For instance,
Path.truediv exists but Path("/home") / None raises TypeError.

So we'd still need to decide whether to mask NAs (auto-propagate) or pass them through. The introspection would tell us which ops to try, but not how to handle missing values in them.

[jorisvandenbossche]

Actually, if both left and right are object dtype, we already do use a masked operation

I think this is a consequence of not having nullable ints/bools and can be deprecated after we are nullable-by-default.

It is probably part of the reason, but we certainly used it for plain object dtype as well (well, notably strings that used object dtype).
Based on failing tests when I disable this code path, at least the following patterns currently make use of this fallback:

• String arithmetic operations with object dtype, e.g. pd.Series(["a", "b", None], dtype=object) * 2 (this was fundamental to our "string support" up to pandas 3)
• "Empty" all-NaN series with object dtype (and since we now use object dtype instead of float as the default dtype, this is more relevant)
• Various other cases that currently require object dtype: Decimals, boolean with missing values

And of course, in general anyone using object dtype with stdlib or custom objects

In the specific case of Series[object].__op__(Series[object]) i don't think we should be masking, just operate pointwise.

I think if we would deprecate this, we should provide some alternative for the use cases of objects where you don't control them, and so cannot make this work with purely element-wise operations

If we do automatic masking, the only thing you loose is the ability to have the missing value not propagate (by implementing the dunder on your object and having it return something else). Personally I think that is a less common use case than just wanting this to work out of the box.