Instruments
Some arguments are easier to poke than to read. These are interactive versions of claims I have made in the writing and the bets. Each one runs entirely in your browser; nothing leaves the page.
Memory-bandwidth roofline
Is a decode workload memory-bound or compute-bound? Plug in the model, batch size, precision, and GPU, and watch the operating point cross the ridge.
Open the instrument →Activation probe cost
What does running an activation probe at inference actually cost? Compare a linear probe against the forward pass it rides on, by model and probe scope.
Open the instrument →The manifold dial
Hyper-connections make composite forward gain explode with depth. A few Sinkhorn iterations project the mixing matrix onto the doubly-stochastic manifold and bound it. Drag the dial.
Open the instrument →Each instrument is a self-contained widget, also embedded in the post it belongs to. More as I build them.