Model selection under class imbalance

The problem, from first principles

This track has already named two separate ways class imbalance breaks the default tools: 02-classification-metrics's Theory notes plain accuracy hides a 95%-negative dataset's failure to catch positives at all, and 11-threshold-optimization showed the default 0.5 threshold is rarely right for imbalanced data. Model selection (choosing between models, or choosing hyperparameters via 04-grid-search/12-nested-cross-validation) has two more of its own defaults that quietly break the same way.

From theory to code

Implement balanced_accuracy(labels, predictions) and stratified_k_fold_split(labels, k, seed=None). Neither changes the model being trained at all — they change what "good" means when comparing candidates, and how "different subsets of the same data" get constructed, the same two places 04-grid-search/12-nested-cross-validation already plug in a score_fn and a fold-splitting function.

Constraints

balanced_accuracy returns the mean of each distinct class's own recall — one vote per class, regardless of class size.
stratified_k_fold_split returns k (train_idx, val_idx) pairs, where every fold's class proportions match the overall dataset's proportions.
Every sample appears in exactly one validation fold; train and validation indices never overlap within a split.

Hints

Open one at a time. Each gives away a little more than the last.

Hint 1

balanced_accuracy computes one recall value per class (mean(predictions[labels == c] == c)), then averages those — never a single pooled accuracy over all samples at once.

Hint 2

stratified_k_fold_split runs 01-splitting-and-resampling's own splitting idea separately per class first (shuffle each class's own indices, split into k pieces with np.array_split), then combines fold i across every class to build that fold's validation set.