In-Context Prediction

The problem, from first principles

In-context prediction

Every single model in this curriculum before this track needed a training loop specific to this dataset: 05-training-loop's gradient descent, 04-full-boosting-loop's sequential tree fitting, 05-em-algorithm's E/M iterations, all of them adjust parameters to fit the data they're given, one dataset at a time. TabPFN's actual, genuinely different idea is in-context learning: train one attention model's weights once, offline, on a huge variety of synthetic datasets, then freeze those weights completely. For any new dataset, no further training happens at all, the training examples themselves become part of the input, laid out as extra rows in the table, and one forward pass through the frozen network produces predictions for new rows directly.

Why there is no positional encoding

A sequence Transformer (the kind Part 2's sequence-modeling questions build toward) has a real problem plain attention doesn't solve on its own: scaled_dot_product_attention (01-row-wise-attention) treats its inputs as an unordered set, nothing in query @ key^T depends on which position a token came from, so "the cat sat" and "sat cat the" would produce identical attention computations unless something explicitly tells the model about word order. A data table has no equivalent problem: row 5 of a dataset isn't "after" row 4 in any meaningful sense, shuffling every row of a training set (and its labels along with it) describes the exact same dataset.

From theory to code

In-context prediction

Implement build_incontext_table(train_features, train_targets, query_features) and in_context_predict(train_features, train_targets, query_features, row_weights, col_weights). Reuse 03-two-way-attention-block's two_way_attention_block.

Why there is no positional encoding

Implement is_row_permutation_equivariant(table, row_weights, col_weights, permutation): checks that shuffling input rows and shuffling output rows are interchangeable. Reuse 03-two-way-attention-block's two_way_attention_block.

Constraints

In-context prediction

• build_incontext_table returns shape (n_train + n_query, n_features + 1, 1): training rows carry their real target in the last column; query rows get exactly 0 there (masked, not a guess).
• in_context_predict returns shape (n_query,): the query rows' target-column values after exactly one forward pass through two_way_attention_block.
• row_weights/col_weights are fixed, given inputs, never updated inside in_context_predict, no training loop of any kind.

Why there is no positional encoding

• Runs two_way_attention_block on table and on table[permutation] (the same weights both times).
• Returns True iff the permuted-input output equals the original output with the same permutation applied to its rows.
• Returns a plain Python bool, not a NumPy boolean.

Hints

In-context prediction

Open one at a time. Each gives away a little more than the last.

Why there is no positional encoding

Open one at a time. Each gives away a little more than the last.