Shadow Deployment: Running a New Model Silently Alongside the Live One
The problem, from first principles
04-canary-deployment exposes a slice of REAL users to a new model: which is safer than a full rollout, but still carries real risk, since canary users genuinely experience the new model's behavior. Shadow deployment removes that risk entirely: run the new model on every request too, but NEVER show its output to anyone or act on it: purely to observe how it WOULD have behaved, with zero user-facing exposure.
From theory to code
Implement shadow_deploy (running both models, returning only the live one's output), outputs_agree (a tolerant comparison), and compute_agreement_rate, quantifying how often the shadow would have matched the live model.
Constraints
shadow_deploy(request, live_model_fn, shadow_model_fn)returns{"response", "shadow_output", "shadow_error"}:"response"is ALWAYS the live model's output.- A shadow model exception must be caught and recorded in
"shadow_error", never allowed to affect"response"or propagate. - A LIVE model exception, in contrast, must propagate normally: a broken live model is a real incident, not something to silently mask.
outputs_agreecompares numeric outputs within a small tolerance, everything else with exact equality.
Hints
Open one at a time. Each gives away a little more than the last.
Hint 1
The live model call and the shadow model call need genuinely DIFFERENT error-handling: the live one should raise like any normal function call (it's a real production incident if it fails), while the shadow one needs to be wrapped in its own try/except so a broken CANDIDATE model never takes down the real response.
Hint 2
compute_agreement_rate is a straightforward application of outputs_agree across a whole list of (live, shadow) pairs: no new comparison logic needed, just averaging the boolean results.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.