Threshold optimization for imbalanced classification

The problem, from first principles

Every classifier that outputs a probability or score, logistic regression's sigmoid, a tree's leaf fraction, needs a threshold to turn that score into an actual 0/1 decision: predictions = scores >= threshold. 0.5 is the default nearly every library reaches for, and it's a completely arbitrary choice, nothing in the model's math requires it.

0.5 is a particularly bad default on imbalanced data. With, say, 5% positive examples, a model can predict "negative" for almost everything and still look fine at that threshold, because there simply aren't many positive examples for a wrong threshold to visibly hurt. But nothing says the threshold has to stay fixed: it's a hyperparameter exactly like 04-grid-search's lr or depth, and it can be swept and optimized the same way, against whatever metric actually matters for the problem, F1, balanced accuracy, or something task-specific, rather than left at a default nobody chose with this dataset in mind.

From theory to code

Theory frames the threshold as a hyperparameter to sweep: try candidate thresholds, score the resulting predictions, keep the best. Implement f1_metric(labels, predictions), a small wrapper reusing 02-classification-metrics's precision_recall_f1, and optimize_threshold(labels, scores, metric_fn=f1_metric), which runs that sweep over the scores actually produced by the model.

Constraints

• f1_metric(labels, predictions) -> float: reuse 02-classification-metrics's precision_recall_f1, return only the F1 value.
• optimize_threshold(labels, scores, metric_fn=f1_metric) -> (best_threshold, best_score).
• Candidate thresholds are exactly the distinct values in scores (np.unique(scores)), not an arbitrary fixed grid like 0.0, 0.1, ..., 1.0.
• metric_fn(labels, predictions) -> float: higher is always better.
• predictions passed to metric_fn come from (scores >= threshold).astype(int).
• Return the threshold achieving the strictly highest metric_fn score, and that score itself.

Hints

Open one at a time. Each gives away a little more than the last.