A/B Testing a Model Change, and Rolling Back When It Loses

The problem, from first principles

05-shadow-deployment compares a candidate model's would-be outputs against the live model's, but never measures its effect on real USER OUTCOMES (does it actually convert more? engage more?). A/B testing is the step that does: split real traffic between the existing model (control) and the new one (treatment), measure a real outcome metric for each group, and use an actual statistical test, not gut feeling, to decide whether an observed difference is real or just noise, and whether to roll back.

From theory to code

Implement assign_variant (deterministic, weighted traffic splitting: generalizing 04-canary-deployment's single-percentage routing to several named variants), evaluate_ab_test (a genuine two-proportion z-test), and decide_rollback.

Constraints

• assign_variant(user_id, variant_names, weights) deterministically assigns the SAME user_id to the SAME variant every time, with the population split roughly proportional to weights.
• evaluate_ab_test(...) computes the standard two-proportion z-test comparing control vs. treatment conversion rates, returning (z_statistic, p_value): a POSITIVE z_statistic means control outperformed treatment.
• decide_rollback(z_statistic, p_value, alpha=0.05) returns True ONLY when the result is BOTH statistically significant (p_value < alpha) AND in the wrong direction (treatment underperforming, z_statistic > 0).
• Rolling back must never trigger just because a result is significant: the DIRECTION matters just as much as the significance.

Hints

Open one at a time. Each gives away a little more than the last.