Column-Wise Attention
The problem, from first principles
01-row-wise-attention let cells within one row attend to each other, feature interactions within a single data sample. But a table has a second, equally important axis: a whole column is one feature across every sample, and there's real information in how one sample's value in a column compares to every other sample's value in that same column, is this row's age unusually high compared to the rest of the dataset, is this row's price an outlier for its column.
From theory to code
Implement column_wise_attention(table, w_query, w_key, w_value): same mechanism as row_wise_attention, transposed to attend across rows within a column. Reuse 01-row-wise-attention's scaled_dot_product_attention.
Constraints
table: shape(n_rows, n_cols, d_model). Returns the same shape.- Cell
(i, c)may attend to any other cell in columnc(any row, same column), never a cell in a different column. - Works correctly for a non-square table (
n_rows != n_cols).
Hints
Open one at a time. Each gives away a little more than the last.
Hint 1
np.swapaxes(table, 0, 1) turns (n_rows, n_cols, d_model) into (n_cols, n_rows, d_model), now columns are the leading "batch" dimension and rows are the sequence, exactly the shape row_wise_attention's own mechanism expects.
Hint 2
After running the same projection-and-attention steps on the transposed table, swap the same two axes back before returning, otherwise the output shape won't match the input's (n_rows, n_cols, d_model) layout.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.