BUG: keep groupby apply return type consistent

[Will-thom]

• closes BUG: pandas groupby.apply return type changes depending on number of unique groupkeys for same custom-function #54992
• keep DataFrameGroupBy.apply on the concat path when a single group returns a Series indexed like the selected object
• avoid treating column-indexed reduction results as transform-like when row and column labels overlap
• add a regression test covering the one-group vs multiple-groups return type consistency

Tests:

• docker exec pandas-gh54992-test bash -lc "cd /workspace && python -m pytest pandas/tests/groupby/test_apply.py::test_apply_single_group_returns_series_like_multiple_groups -q"
• docker exec pandas-gh54992-test bash -lc "cd /workspace && python -m pytest pandas/tests/groupby/test_apply.py -q"
• docker exec pandas-gh54992-test bash -lc "cd /workspace && python -m pytest pandas/tests/groupby -q"

[rhshadrach]

Thanks for the PR! If I don't get a review in by June 1st, please ping me.

[rhshadrach]

Thanks for the PR, but I'm leaning negative on introducing more behaviors such as the one being done here. The conditions on the result index equaling / not equaling the index / columns of the input is fragile and quite surprising. E.g. just changing a 2 to 3 below completely changes the result shape.

def foo(x):
    return pd.Series({0: x['b'].sum(), 1: x['b'].mean(), 2: x['b'].std()})

df = pd.DataFrame({'a': [1, 1, 1], 'b': [3, 4, 7]})
print(df.groupby('a').apply(foo))
# a   
# 1  0    14.000000
#    1     4.666667
#    2     2.081666
# dtype: float64

def foo(x):
    return pd.Series({0: x['b'].sum(), 1: x['b'].mean(), 3: x['b'].std()})

df = pd.DataFrame({'a': [1, 1, 1], 'b': [3, 4, 7]})
print(df.groupby('a').apply(foo))
#       0         1         3
# a                          
# 1  14.0  4.666667  2.081666

Now I think one could argue that this case is likely to be uncommon, which is what is keeping me from being strongly opposed to this kind of change, but I still do not think this is the right kind of solution we want.

[Will-thom]

Hi @rhshadrach, gently pinging this as suggested since June 1 has passed. Please let me know if there is anything else needed from my side. Thanks!

[rhshadrach]

@Will-thom - I left my feedback above.

[jbrockmendel]

closing as stale. can reopen when you're ready to address comments and continue