bug: DataSource DataFrame methods silently overwrite input columns when output shares a name

[SudipSinha]

Bug

When input and output tensors share a column name (e.g. "score"), DataSource methods that build a DataFrame from both input and output arrays silently overwrite the input column with the output column. Drift callers then compute drift on output data instead of input data, producing incorrect results with no error.

Ref: RHOAIENG-75329

Root cause

Both get_dataframe_with_batch_size (lines 106–142) and get_dataframe_by_tag (lines 213–232) in src/service/data/datasources/data_source.py merge input and output columns into a single Python dict (df_data). Python dict assignment overwrites on duplicate keys, so the output value silently replaces the input value.

Affected methods

• DataSource.get_dataframe_with_batch_size() — used by get_organic_dataframe(), which feeds all drift and fairness endpoints
• DataSource.get_dataframe_by_tag() — used by drift endpoints for reference data

Affected callers

• kolmogorov_smirnov.py — reference_df[feature_name].to_numpy() gets output data when feature_name collides
• compare_means.py — same pattern
• jensen_shannon.py — same pattern
• approx_ks_test.py — same pattern
• fourier_mmd.py — same pattern
• SPD/DIR fairness endpoints — access columns by name from the same merged DataFrame

Realistic collision scenarios

• Credit scoring: input credit_score (from bureau) + output credit_score (predicted)
• Feature imputation: input score (with nulls) + output score (imputed)
• Autoencoders/VAE: inputs [f1, f2, f3] + outputs [f1, f2, f3] (reconstructed)
• KServe V2 spec does not enforce distinct input/output tensor names

Possible fixes

1. Namespace columns: prefix input columns with input_ and output columns with output_ in the DataFrame. Requires updating all callers to use prefixed names.
2. Return separate DataFrames: change return type to a named tuple (input_df, output_df). Callers access the DataFrame they need. Most natural fix for drift endpoints which only use input columns.
3. Detect and raise: check for collisions before merging and raise ValueError. Simplest fix but breaks existing data that happens to have collisions.

Reproduction

1. Upload a KServe payload where an input tensor and output tensor share the same name (e.g. both named "score")
2. Request a drift metric on that column
3. The drift computation will silently use the output values as the "current" data instead of the input values