Model Degradation Over Time, and Deciding When to Retrain

The problem, from first principles

02-concept-drift-sliding-window detects that degradation has ALREADY happened, by looking back at recent performance. A production team also wants the forward-looking question answered: given how fast a model is currently degrading, how much longer does it have before it needs attention, and, right now, has it already crossed the line?

From theory to code

Implement has_model_degraded (the actual current-state check), predicted_future_metric (a simple linear extrapolation), days_until_degraded (turning that extrapolation into an actionable countdown), and should_retrain_now.

Constraints

has_model_degraded(current_metric, baseline_metric, tolerance) returns True when the metric has dropped by MORE than tolerance below baseline (strict inequality).
predicted_future_metric(current_metric, degradation_rate_per_day, days_ahead) returns current_metric - degradation_rate_per_day * days_ahead.
days_until_degraded(...) returns math.inf for a zero-or-negative degradation rate, 0.0 if already past the threshold, and otherwise the remaining "budget" divided by the rate.
should_retrain_now is built directly on top of has_model_degraded.

Hints

Open one at a time. Each gives away a little more than the last.

Hint 1

The "degradation budget" is exactly how much the metric could still fall before crossing the degraded threshold: current_metric - (baseline_metric - tolerance): divide that by the daily rate of loss to get how many days remain.

Hint 2

If the degradation budget is already <= 0, the model has ALREADY crossed the threshold right now: days_until_degraded should return 0.0 in that case, not a negative number.

Model Degradation Over Time, and Deciding When to Retrain

The problem, from first principles

From theory to code

Constraints

Hints

Keywords