Row-wise attention over table cells

The problem, from first principles

Every model in this curriculum before this track processes a table's rows as fixed-length feature vectors and learns one set of weights that treats every column the same way for every row (a linear model's weight, a tree's fixed splits). TabPFN-style tabular foundation models take a different approach: treat the table itself as the input to a Transformer-style attention mechanism, letting the model learn, per table, which cells should influence which.

From theory to code

Implement softmax(x, axis=-1), scaled_dot_product_attention(query, key, value), and row_wise_attention(table, w_query, w_key, w_value). table represents a whole data table already embedded into vectors: shape (n_rows, n_cols, d_model), one d_model-dimensional vector per cell (one sample's one feature value). row_wise_attention runs attention within each row, across that row's own cells, never mixing information between different rows.

Constraints

• softmax(x, axis=-1) sums to 1 along axis, stays finite for large inputs.
• scaled_dot_product_attention(query, key, value) returns a weighted average of value rows, one output row per query row.
• row_wise_attention projects table with w_query/w_key/w_value (each (d_model, d_model)), then runs attention independently within each row — cell i in row r may attend to any cell in row r, never a cell in a different row.

Hints

Open one at a time. Each gives away a little more than the last.