Missing Values: Measure, Fill, Drop

The problem, from first principles

Real tables have holes: a sensor that dropped a reading, a form field left blank, a join that found no partner. pandas marks a hole with NaN (or None, or NaT for dates), and what you do about it is a modelling decision with consequences. Dropping rows throws information away, and can bias the sample if the holes are not random. Filling with the overall average hides real differences between groups. Filling a time series with the last known value is reasonable for a stock price and nonsense for a count of events.

Before choosing, measure: how many holes, in which columns? This question builds the measuring tool, three fill strategies with different assumptions, and the habit that makes imputation honest: remembering which values were filled.

From theory to code

Implement missing_report(df), fill_group_median(df, key, column), forward_fill_limit(s, limit), drop_sparse_columns(df, max_missing_fraction) and fill_with_indicator(df, column). The signatures and docstrings are already in the editor.

Constraints

• missing_report(df) returns a frame indexed by column name containing only columns with at least one missing value, with the columns n_missing (integer count) and pct (100 * n_missing / len(df) rounded to 1 decimal). Rows are ordered by n_missing descending, ties by column name ascending. If nothing is missing it returns an empty frame with those two columns.
• fill_group_median(df, key, column) returns a float Series (index of df) equal to df[column] with each missing value replaced by the median of the known values of its own group (rows sharing the same key). If the group has no known value, or the row's key is missing, use the median of the whole column.
• forward_fill_limit(s, limit) carries the last known value forward into missing positions, but fills at most limit consecutive positions after each known value; later gaps stay missing. Leading missing values stay missing.
• drop_sparse_columns(df, max_missing_fraction) returns df without the columns whose fraction of missing values is greater than max_missing_fraction (equal is kept). Column order is kept.
• fill_with_indicator(df, column) returns a copy in which column's missing values are replaced by the column median and a new boolean column <column>_was_missing (placed last) records which rows were filled. The input is not changed.

Hints