Activation Patching

The problem, from first principles

A probe shows that information is present; it does not show the model uses it. To test causal importance we intervene. Run the model on a clean prompt where it answers correctly and on a corrupted prompt where it answers wrongly. Then rerun the corrupted prompt but patch in one activation copied from the clean run. If that single patch restores the correct behavior, the activation carries information the answer depends on. Scanning every position or component gives a map of where the computation happens. Results are reported as a recovery fraction: 0 means the patch changed nothing and 1 means it fully restored the clean behavior.

From theory to code

Implement patch_rows, recovery and patching_scan.

Constraints

• patch_rows(clean, corrupt, rows) returns a copy of the corrupted activations corrupt (shape (T, d)) with the listed rows replaced by the same rows of clean.
• recovery(clean_metric, corrupt_metric, patched_metric) is (patched - corrupt) / (clean - corrupt).
• patching_scan(model, clean, corrupt) calls model(acts) (a function from a (T, d) array to a scalar metric such as a logit difference) once on the clean activations and once on the corrupted ones, then once per position t with only row t patched. It returns the array of recovery fractions, one per position.
• Do not modify clean or corrupt.

Hints