Data Drift: The Input Distribution Shifting After Deployment

The problem, from first principles

06-ab-testing-rollback compares two MODELS against a fixed traffic source. But the world a deployed model sees keeps changing on its own, independent of any model change at all: customer demographics shift, a new product launch changes typical purchase amounts, a sensor gets recalibrated. Data drift is exactly this, the INPUT distribution a model sees in production drifting away from the distribution it was trained on, and it needs its own real, quantifiable detection metric, separate from anything about model accuracy.

From theory to code

Implement bin_distribution (turning raw samples into a normalized histogram) and population_stability_index (PSI), the standard industry metric for quantifying distribution shift, plus detect_data_drift.

Constraints

• bin_distribution(samples, bin_edges) returns a normalized histogram summing to 1.0, with any empty bin's zero count replaced by a tiny floor value (1e-6) before normalizing.
• population_stability_index(expected, actual) returns sum((actual - expected) * ln(actual / expected)) over two ALREADY-NORMALIZED distributions.
• detect_data_drift(psi_value, threshold=0.2) is a plain threshold comparison.
• PSI must be near 0 for two samples drawn from the SAME true distribution, regardless of sample size, and large for a genuinely shifted distribution.

Hints

Open one at a time. Each gives away a little more than the last.