Activation probe cost
What does running an activation probe at inference actually cost? Compare a linear probe against the forward pass it rides on, by model and probe scope.
Bet 12 / Runtime interpretability
What does an activation probe actually cost?
A linear probe is one dot product against a hidden state the forward pass has already computed. Set next to the forward pass itself, the arithmetic is a rounding error. The real cost is engineering, not FLOPs.
How this is computed
Per generated token: forward pass ≈ 2 × params FLOPs. A linear probe on the residual stream is 2 × d_model FLOPs for one layer, or 2 × layers × d_model to probe every layer.
This is the compute argument only, and it is the weakest objection to always-on probing. The costs that actually matter are engineering: where you tap activations, how you batch the reads, and keeping the probe off the critical serving path. That is the whole point of the post.