Stretch: regularized boosting (shrinkage + L2 leaf penalty, XGBoost-style)
Advanced classical-mlensemblesgradient-boostingregularizationstretch
Implement:
def xgboost_leaf_value(gradients, hessians, lam) -> float: ...
def xgboost_split_gain(gradients, hessians, left_mask, lam) -> float: ...
def find_best_regularized_split(input, gradients, hessians, lam) -> tuple[int, float, float] | None: ...
def build_regularized_tree(input, gradients, hessians, max_depth, lam) -> dict: ...
def train_regularized_boosting(input, targets, n_trees, max_depth, learning_rate, lam) -> tuple[float, list[dict]]: ...
def predict_regularized_boosting(initial_prediction, trees, learning_rate, input) -> np.ndarray: ...
- Reuse
05-regression-trees's predict_regression_tree for prediction, tree traversal doesn't change.
- Squared-error loss throughout, so hessians are constant (
1.0 per sample).
lam=0 should behave almost identically to 04-full-boosting-loop's plain gradient boosting, this is a real sanity check, not a coincidence.