Code: github.com/bassrehab/api-mcp-compiler · Docs: api-mcp.subhadipmitra.com · Install: pip install api-mcp-compiler

TL;DR: pip install api-mcp-compiler compiles OpenAPI 3.x, Swagger 2.0, and WSDL 1.1 into MCP tool surfaces that are designed rather than transliterated: tools named for tasks, transport plumbing and deprecated operations dropped, lookup-then-act sequences composed. Every field carries provenance back to the source. Governance is a separate reviewable manifest, with narrowest sufficient scopes and rate budgets, and destructive tools are emitted disabled until a person approves them by name, group, or risk class. Approvals are bound by digest to the exact revision of the spec, so they do not silently survive a change to it.

Every gateway vendor now ships a button that turns an OpenAPI spec into an MCP server. Press it and your 87-operation service becomes 87 tools, each one a faithful transliteration of an endpoint that was designed for a frontend team in 2019. The conversion is real, the demo works, and the hard problem has not been touched.

The hard problem is deciding which tools should exist at all, what they should be called, what they should accept and return, and which of them must never run without a human saying so. That is interface design, and the interface’s consumer is a language model choosing its next action from names and descriptions. I spent the past few months building a compiler around that problem: api-mcp-compiler, which takes OpenAPI 3.x, Swagger 2.0, or WSDL 1.1 and produces an MCP tool surface with provenance on every field, a governance manifest as a separate reviewable artifact, and destructive operations compiled in disabled until a person approves them.

I will also show you the part that did not go the way I expected. I pre-registered four evaluations to prove that a semantically planned surface beats naive one-tool-per-operation, and all four came back underpowered. I think the numbers are more useful than the claim would have been, and I would rather report them precisely than bury them. But the reason the project exists is the governance half, and that half was never on trial.

The transliteration trap

Take a real specification and convert it operation by operation. Here is what the agent inherits:

• Colliding names. One API I ingested had ten operations whose summaries all read “Get Details.” An agent selects tools by name and description. Ten tools that describe themselves identically is not a tool surface, it is a lottery.
• Transport plumbing promoted to interface. Pagination cursors, response envelopes, and format toggles become tool arguments the model is invited to guess at, even where the server has perfectly good defaults.
• Dead surface. Deprecated operations arrive with the same standing as live ones, spending the model’s attention on endpoints the provider intends to withdraw.
• Destructive parity. DELETE /accounts/{id} shows up enabled, right next to the harmless reads, because the spec never said which operations should make a reviewer nervous. Specs describe capability, not intent.

None of this is the spec’s fault. An OpenAPI document is a contract between machines that already know what they want. An agent tool surface is an interface for a consumer that decides what it wants by reading the interface. Those are different artifacts, and I have come to think the second should be compiled from the first the way object code is compiled from source: through explicit, inspectable stages, not string substitution. I have written before about where organizations sit on context maturity and how the protocol stack is settling; this project is what I think the bottom layer of that stack should look like when it grows up.

The axes that separate the two artifacts, stated plainly. I am not naming products, because this is a difference of category rather than a scoreboard, and any specific tool can close any specific row tomorrow:

A compiler, not a converter

The pipeline has five stages, each emitting an artifact with its own versioned JSON Schema contract, so a change in one stage cannot be silently misread by the next.

flowchart LR
    A[OpenAPI / Swagger / WSDL] --> B[Ingestion<br/>Semantic IR]
    B --> C[Planning<br/>proposals + overlay]
    C --> D[Policy<br/>governance manifest]
    D --> E[Generation<br/>MCP server]
    E --> F[Evaluation<br/>state oracles]

Ingestion is deliberately paranoid. No network calls happen during it, remote $ref resolution is denied, and file loads are restricted to explicitly allowed directories. Every field in the resulting IR carries provenance: where in the source it came from, how it was derived, and with what confidence. Nothing is silently dropped. A construct the compiler cannot resolve becomes an explicit Ambiguity record, a first-class output rather than a warning scrolling past in a log.

A document that splits its schemas across URLs still compiles, because fetching is a separate command rather than a flag on ingestion. vendor-refs fetches each remote document once, pins it by digest and writes a lock; a compile then reads the pinned files, verifies them, and refuses anything the lock does not name. Ingestion still performs no network access, and an upstream edit surfaces as a digest mismatch rather than as a surface that quietly became something else.

The whole flow is a CLI:

python -m api_mcp_compiler.cli inspect  examples/openapi/inventory_service.yaml  # IR, side effects, provenance
python -m api_mcp_compiler.cli review   examples/openapi/inventory_service.yaml  # planner proposals, as markdown
python -m api_mcp_compiler.cli policy   examples/openapi/inventory_service.yaml  # governance manifest
python -m api_mcp_compiler.cli generate examples/openapi/inventory_service.yaml  # surface, destructive tools disabled
python -m api_mcp_compiler.cli report   examples/openapi/inventory_service.yaml  # self-contained HTML approval doc
python -m api_mcp_compiler.cli approve  examples/openapi/inventory_service.yaml \
    --overlay build/inventory.overlay.json --risk destructive                    # decisions land in an overlay
python -m api_mcp_compiler.cli serve    examples/openapi/inventory_service.yaml \
    --overlay build/inventory.overlay.json --out build/inventory_server.py      # runnable MCP server

The planner proposes, you dispose

The planning stage looks at the IR and proposes changes to the baseline one-tool-per-operation mapping. Seven kinds of proposal exist:

1. Rename: name tools around tasks instead of handler methods, so listWarehouseItems becomes list_items_held_warehouse, derived from the operation’s own summary.
2. Group: group by the coarsest structure the specification actually states (the first path segment), not a taxonomy the planner invents.
3. Project: drop optional transport arguments that have server defaults, pagination cursors being the classic case.
4. Describe: rewrite descriptions for a model reader, stripping web formatting and foregrounding actual side effects.
5. Reclassify: turn addressable reads into MCP resources so lookups stop consuming tool slots.
6. Omit: drop deprecated operations, so agent attention is not spent on surface the provider intends to withdraw.
7. Compose: propose workflow tools for lookup-then-act sequences.

An operation answering 202 has been accepted, not performed. A surface silent about that lets an agent read acceptance as completion, report the goal met, and move on before the work has started, and every part of that failure looks like success. The compiler records it and the tool description says so, naming the header to poll where the document declares one and saying plainly that it does not where the document is silent.

Every proposal is recorded with a kind, a target, a rationale written for a person, and a confidence derived from signals in the specification rather than the planner’s self-regard.

The important part is what the planner does not get to do: decide. A plan is a set of proposals. Decisions live in an overlay file that the approve command generates, bound by digest to the exact revision of the specification. When the spec changes, the approvals do not silently carry over. Nobody hand-edits JSON, and the tool surface never shrinks without a recorded reason.

Easier to operate than to describe. Below is one specification converted both ways, with the panes fed by an actual compiler run rather than an illustration: each side is generate over the same document, once with the baseline planner and once with the semantic one, and the rationales, confidences, and governance are the records the compiler itself emitted. Pick any operation to see what changed and why.

Tools that earn the right to execute

Generation runs everything through an emission gate. The docs state the rule plainly: a tool is emitted executable only if it earned it. Reads pass automatically once validated, because a reviewer clicking through twenty-three read tools one at a time is doing data entry, not governance. Writes and destructive operations need a recorded approval, granted by risk class, by group, or by individual name.

Everything that fails the gate is still emitted, but disabled, carrying the reason it was refused. This sounds like a small choice and is not: a surface that silently omitted an operation would be indistinguishable from one where the operation never existed, and nobody reviews an absence. The refusal is compiled into the artifact, not enforced by a runtime configuration flag that someone can quietly flip in production.

Two more properties do a lot of the safety work:

• Inference is safety-monotone. Analysis may reclassify an operation as more restricted than the spec suggests, never less. A write that cannot demonstrate its authorization story fails closed.
• Confirmation does not transfer, and does not keep. An approval token for a destructive call is bound to a digest of the specific arguments and is single use with an expiry: a lapsed token is refused and a fresh confirmation is required (TTL enforcement shipped in 0.2.0). A confirmation for one operation cannot be replayed to authorize a different one.

Alongside the code, the policy stage emits a per-tool governance manifest: narrowest sufficient scopes rather than a union of alternatives, approval class, rate budgets (calls per minute, concurrency, daily budget), and rollback guidance for destructive operations. Policy is a separate artifact precisely so that the people who review governance never have to read generated code to find it.

Separating the manifest from the code has a failure mode I walked into. Four values were derived by the policy manifest, written into every generated server, and acted on by nothing. The confirmation time to live above is one of them: tokens never expired. Credential placement was another, and the compiler had parsed the correct placement and discarded it at the last step, so every scheme was sent as Authorization: Bearer and an API key in a service-named header, or HTTP basic, produced a 401 on every call. A retry policy of never retried nothing only because nothing retried at all, which looks identical to compliance from the outside. And a destructive tool carrying a budget of two calls a minute would make two hundred. All four have been fixed since this post first went up.

All four passed every test, because every test of generated code asserted on its source text. requires_confirmation and token_ttl_seconds were both provably in the file. Reading a file proves a value was written into it; it never proves anything reads it. The tests now execute the emitted module and inspect the request it actually makes, and that change is what found all four.

The SOAP path is not a footnote

If you have a legacy estate, this is the part you actually care about. WSDL 1.1 is a first-class input, not a translation layer over the REST pipeline: namespaces, XSD types, SOAP headers, faults, and bindings all survive ingestion. XSD types translate to JSON Schema with their facets intact, including pattern, enumeration, and length, and minOccurs and maxOccurs cardinality. Document/literal and rpc/literal bindings are both ingested. Section 5 encoding is refused rather than approximated, because it serializes values as a reference graph the compiler does not write.

SOAP also changes what inference is allowed to do. WSDL carries no signal equivalent to an HTTP method, so a SOAP operation is never classified by inference: every one arrives unclassified, and the emission gate blocks it until a reviewer records the side effect. This is deliberate. An operation named DeleteCustomer might delete, and might return a receipt.

The path is verified against 40 WSDL documents from the public SAWSDL test collection, fetched and never redistributed, and against two live public SOAP services that answered real calls through generated servers.

The part I could not prove

The obvious question: does any of the planning actually help the agent? I assumed the answer was yes and set out to demonstrate it. Because “assumed the answer was yes” is exactly the condition under which people fool themselves, the harness enforces pre-registration. Before a model-backed comparison runs, the hypothesis, corpus, arms, model, success criteria, budget, primary statistical test, threshold, and falsification conditions are written down and digested. Runs record the digest of their registration, the harness refuses to combine runs whose digests differ, and registrations are append-only. A result cannot be attached to a hypothesis written after the fact, including by me.

Scoring is deterministic. A task succeeds if the service ends in the state the goal asked for, judged by an oracle against the final store, not by a model grading a transcript. Oracles name source operations rather than tool names, so one corpus scores both arms without favoring either. The primary test was McNemar’s exact test, two-sided, alpha 0.05, which on a 24-task corpus needs at least six discordant pairs, all favoring the same arm.

Four registrations ran to completion:

Let me be precise about what the table says. It is not evidence that planning works: every point difference is one discordant pair against a threshold of six. It is also not evidence that planning is useless: an underpowered test that fails to reject tells you about the test, not the world. At the discordance rate the runs actually observed, a 55-task corpus expects about 2.3 discordant pairs, and roughly 150 tasks would be needed to expect six. These corpora were never going to answer the question, and the specs where I expect design to matter most, the 400-operation enterprise WSDLs with a decade of accretion, are exactly the ones that are never public.

If I had run one comparison, gotten 14 against 13, and skipped the registration step, this section would be a triumphant chart. Pre-registration is the reason it is a table with “inconclusive” written four times, and I have come around to thinking that is the more useful artifact.

The most interesting measurement in the project is hiding in the TMDB row. Both arms scored 28 of 34 with zero discordant pairs, but they did not get there the same way. The agent reached for a composite tool on 29 of 34 tasks, 18 of the 29 composites the plan offered were used at least once, and total calls fell from 312 to 289, a reduction of 7.4 percent. Context bytes rose about 20 percent, because a composite returns the last step’s payload while the baseline agent often stopped at the smaller one it needed. That is a finding: composition changed what the agent did without changing what it achieved, and the trade it made was calls for context. (Secondary, and it stays secondary: the semantic arm used fewer calls on 17 tasks and more on 9, a two-sided sign test at p = 0.169. Suggestive, not evidence.)

What the harness caught instead

The evaluations refused to resolve the headline hypothesis, and paid rent anyway. Building oracles that check final state instead of model behavior turned the eval harness into an instrument, and the instrument found defects the demos never would have:

• Redaction deleted a live service’s answer because the field name TitleCaseWordsWithTokenResult contains the word “token”. The tool returned [redacted] with HTTP 200 and no hint that anything had been withheld. Sensitivity is now decided on the noun a field name ends on, not on any overlap with a secret vocabulary.
• A generated SOAP body used the message part’s own name instead of the element the part references. Every local fixture passed. Every real service returned a fault.
• 22 of 40 generated tools carried schemas that no JSON Schema validator would accept, because the source specification writes "maximum": "50" as a string: valid JSON, an invalid schema, and invisible until something checked it mechanically.

This is the general lesson I keep relearning, most recently while building eval loops for verifiers: deterministic evaluation infrastructure is worth building even when the experiment is inconclusive, because the harness audits everything it touches.

And it is why the inconclusive table does not undermine the project. Provenance, digest-bound approvals, fail-closed writes, and non-transferable confirmation tokens were never claims about task success rates. They are claims about what an agent may do without a human, and they hold whether semantic planning turns out to be a win, a wash, or a rounding error.

What this is not

The compiler documents its own limits, and I will repeat them here. A generated artifact cannot provide server-side authorization, protection against confused-deputy designs, or end-user identity propagation. Those are properties of the deployed service, and a tool surface that claims to solve them from the client side is lying to you. The evaluation corpus is two public APIs and small task sets, which is a floor, not a ceiling.

Try it

The repo is Apache 2.0, typed strictly, and ships with an executable notebook, golden artifacts for reproducibility, and the full docs, including the evaluation methodology with every registration public. It is on PyPI:

pip install api-mcp-compiler

Two things would genuinely move this forward. First, larger task corpora: the arithmetic above says roughly 150 tasks, and that is breadth I cannot generate alone. Second, ugly specifications. The hypothesis that tool surfaces should be designed rather than transliterated is at its most plausible on large, old, colliding surfaces, and those never appear in public benchmarks. If you have a WSDL from 2009 that you are willing to share even in redacted form, I would like to compile it.

Code: github.com/bassrehab/triton-kernels · Kernel: huggingface.co/kernels/bassrehab/w4a16

Last time I fused an entire Mixture-of-Experts forward pass into Triton and it beat Megablocks. That kernel went after the routing. This one goes after the other half of the memory-bound inference problem: the weights.

LLM inference is bottlenecked on weight loading. A 7B model in FP16 is 14GB that you drag across the memory bus on every forward pass. Quantize those weights to 4 bits and you move a quarter of the bytes. That is the entire pitch behind GPTQ and AWQ, and it is why 4-bit weight-only quantization is how most people actually deploy LLMs in 2026.

There is a catch. The fast 4-bit kernels, Marlin and AWQ and exllama, are all CUDA. If you run on AMD, or you just do not want to bet your inference stack on a single vendor, you are stuck.

So I wrote a W4A16 GEMM in pure Triton. FP16 activations, 4-bit weights, dequantized inside the kernel. It runs on NVIDIA and AMD with no CUDA and no vendor intrinsics.

Here is the honest version of the result. My first correct kernel was three times slower than a plain FP16 matmul. Portable, numerically perfect, and completely useless. Getting it to actually beat FP16 took two rewrites and taught me more about memory bandwidth than the MoE kernel did. The final version beats cuBLAS FP16 by 1.1 to 1.3x in the decode regime, which is exactly the regime that matters when you generate tokens one at a time.

Why 4-bit weight-only is the right lever

When you decode text, you produce one token at a time. Batch size is tiny, often one. At that size the GPU has almost nothing to compute, so the whole forward pass is a race to read weights out of HBM. The math is nearly free. The memory is everything.

This is the memory-bound regime, and it is where quantization pays off. Four-bit weights are a quarter the size of FP16, so you move a quarter of the bytes. If the kernel were perfectly bandwidth-bound, that would be a clean 4x.

One number kept me honest the entire time. At batch size 1, cuBLAS FP16 already hits about 71% of the A100’s peak memory bandwidth. That is not a lazy baseline. To beat it you have to move less data and keep the pipes just as full. Moving less data is trivial. Keeping the pipes full while unpacking 4-bit weights on the fly is the hard part, and it is where all the difficulty lives.

The design

The weights are quantized to 4 bits with one scale and one zero-point per group of 128 rows, GPTQ and AWQ style. Eight 4-bit values pack into a single int32 along the contraction dimension. Dequantization happens inside the GEMM loop, so the weights cross the memory bus at 4 bits and only expand to FP16 once they are already on chip, in registers.

The inner loop is the whole kernel:

# Load one packed int32 (8 weights), unpack all 8 nibbles in registers,
# dequantize with the group's scale, then matmul. No 4-bit weight is read twice.
pk = tl.load(b_ptrs)                                    # (BLOCK_K // 8, BLOCK_N) int32
w  = (pk[:, None, :] >> shifts[None, :, None]) & 0xF    # (BLOCK_K // 8, 8, BLOCK_N)
w  = tl.reshape(w, (BLOCK_K, BLOCK_N)).to(tl.float16)   # row 8*i + j maps to K = 8*i + j
w  = (w - zero) * scale                                 # dequant, still in registers
acc += tl.dot(x, w, out_dtype=tl.float32)               # FP16 tensor cores, FP32 accumulate

That “no 4-bit weight is read twice” comment is doing a lot of work. It is the single change that took the kernel from useless to competitive, and I will come back to why in the war-stories section.

The other half of the design is a split-K path for decode. A skinny matmul, one or a few rows tall, launches very few thread blocks, so most of the GPU sits idle and you never saturate HBM. Splitting the contraction dimension across many blocks, each computing a partial sum and accumulating into the output, brings the parallelism back. The kernel dispatches to split-K for small batches and to the plain tiled kernel once the batch is large enough to fill the machine on its own.

Results

Measured on an A100-SXM4-80GB against torch FP16, which is cuBLAS underneath. Group size 128, on current-generation models. Speedup is W4A16 over FP16 at decode batch sizes.

The kernel beats FP16 across the decode regime on both the dense FFN and the MoE-expert projections, and it does it while using a quarter of the weight memory. The speedups track each other closely across models from 2024 to 2026, because a weight-only GEMM is set by its matrix dimensions and those have barely moved between generations. Once the batch grows past a handful of tokens, into prefill, the unpack overhead stops hiding behind the weight traffic and cuBLAS pulls back ahead. That crossover is fine. Prefill runs once per request; decode runs once per generated token. Decode is where the bill is.

The roofline also shows what I did not manage. Peak achieved bandwidth on the weight stream tops out around 24% of the A100’s HBM. I am collecting maybe a third of the 4x the format promises; most of it is still on the table. More on that in what’s next.

Things I got wrong along the way

The 8x redundant load. My first kernel indexed the packed weights by offs_k // 8, which meant eight consecutive rows of the contraction dimension all loaded the same int32. Read the same word eight times and you have moved eight times the packed data. That is more traffic than FP16, not less. The kernel was correct and roughly three times slower than a plain matmul, achieving about 4% of peak bandwidth. The fix is the loop above: load each int32 once, unpack all eight nibbles in registers. Obvious in hindsight, and it took me an afternoon and a bandwidth counter to see it.

Parity is not a win. After the load-once fix the kernel reached parity with cuBLAS on the large matmuls and stayed slower on the small ones. Correct, portable, and exactly as fast as the thing it was supposed to beat. The bandwidth counter said 19% of peak, which meant the kernel was no longer memory-bound at all. It was overhead-bound, spending its time on the unpack and the register shuffle rather than on memory. Beating FP16 needed a different lever entirely, and that lever was parallelism, not bandwidth.

Split-K only helps when the matmul is skinny. Adding split-K got me past FP16 at batch 1. It also made the kernel slower at batch 32, because the extra blocks each write a partial sum through an FP32 atomic add, and once there is enough work to fill the GPU that atomic traffic costs more than the parallelism buys. So split-K is a dispatch decision, not a default.

The best split factor is not monotonic in the shape. I assumed a bigger contraction dimension wanted more splits. Wrong. The attention projection wanted eight splits, the FFN wanted four, and they have the same contraction dimension. The optimum depends on the full shape in a way I could not capture with a clean rule, so I stopped trying to be clever and handed it to triton.autotune. The subtlety: autotuning a kernel that accumulates through atomic add corrupts its own output during the timing trials, so you have to pass reset_to_zero to re-zero the accumulator before each trial. Miss that and every autotuned result is garbage.

Naming the file after the package. This one is embarrassing. When I packaged the kernel for the Hugging Face Kernel Hub I put the code in w4a16/w4a16.py inside a package also named w4a16. The build’s import check failed with a circular import, because from .w4a16 import ... resolved to the half-initialized package instead of the module. Renamed the file to gemm.py and it built. Twenty minutes of staring at a stack trace that was entirely my fault.

What’s next

The 4x is still out there. Collecting it means a Marlin-style layout: pre-permute the weights offline so they feed the tensor cores in exactly the order the MMA wants them, without the runtime reshape, and pipeline the loads with async copies so the unpack hides under memory latency instead of competing with it. That is a real project, not a tweak, and a Triton port probably lands somewhere short of Marlin’s hand-tuned CUDA. But it should turn the decode win from 1.3x into something a lot bigger.

Also on the list: FP8 activations for a W4A8 variant, and correctness validation on an AMD MI300X to make the cross-platform claim concrete rather than structural. The kernel is pure Triton with no vendor code, so it runs on AMD by construction. I just want to prove it on the hardware.

Code

Everything is in the repo:

• triton_kernels/w4a16.py: the kernel, the split-K path, and the weight-packing utilities.
• reference/w4a16_reference.py: the plain-PyTorch ground truth.
• tests/test_w4a16.py: correctness across shapes, group sizes, and symmetric and asymmetric quantization.
• benchmarks/roofline/w4a16_roofline.py: the roofline plot.
• docs/w4a16.md: the full writeup.

Or load it straight from the Hub, no install:

from kernels import get_kernel

w4a16 = get_kernel("bassrehab/w4a16", version=1, trust_remote_code=True)
packed, scales, zeros = w4a16.quantize_weight_int4_grouped(weight, group_size=128)
y = w4a16.w4a16_gemm(x, packed, scales, zeros, group_size=128)

Takeaways

1. The naive 4-bit kernel is a trap. It is easy to write a correct W4A16 kernel that moves more memory than FP16, not less. Watch the bytes you actually move, not just the output. If your weight traffic is not below FP16’s, you have bought nothing, you are just paying for the unpack.

2. Decode is a parallelism problem, not only a bandwidth problem. At batch 1 the bottleneck is that the GPU is empty, not that the bus is slow. Split-K fills it. That was the counterintuitive part for me: the fix for a memory-bound kernel was more parallelism.

3. Stop hand-tuning shapes. When the optimum is non-monotonic in the shape, a heuristic will fight you forever. Autotune it, and remember reset_to_zero if you accumulate through atomics.

4. Portable and competitive are compatible, but the last 4x is CUDA’s home turf. Pure Triton got me a real decode win over cuBLAS on any vendor. Matching Marlin is a separate, harder fight, and an honest kernel says so.

TL;DR: I evaluated five jailbreak-defense paradigms across seven instruction-tuned models (7B-31B) and five attack families. The headline: activation-based defenses that act at prompt time are structurally blind to prefilling attacks - not weakly, not sometimes, but by construction. AlphaSteer stops GCG and AutoDAN completely, then lets prefilling through at ~50%. The fix turned out to be embarrassingly simple: read the hidden states of the first generated tokens instead. A linear probe there hits AUROC 0.97-1.00 on every model I tested, and paired with a halt rule it drives prefilling success to zero with no false positives on benign inputs. Paper and full artifacts: arXiv:2606.29441.

Where this came from

My red-teaming work has been attack-side for a year: quality-diversity evolution that discovers diverse jailbreaks automatically, and the follow-up finding that safety alignment moves non-monotonically across model generations. Somewhere in the middle of building attack archives, the obvious question flipped around: the defense literature keeps reporting near-perfect numbers, so why do deployed models keep getting jailbroken?

The uncomfortable answer is that defenses get evaluated against the attacks they were designed around. GCG suffixes, AutoDAN prompts, roleplay nesting - these all do their damage inside the prompt, and that’s exactly where modern activation-level defenses look. So I ran the cross-product instead: five defense paradigms, five attack families, seven models. The matrix has a hole in it, and the hole has a shape.

The blind spot is geometric, not empirical

Activation defenses like AlphaSteer work on a simple, elegant premise: harmful requests land in a recognizable region of activation space - call it the harmful cone - and you can gate or steer based on where the prompt’s representation sits relative to that cone.

Prefilling attacks never enter the cone. Instead of persuading the model to comply, they put words in its mouth: the attacker seeds the beginning of the assistant’s response (“Sure, here’s how to…”) and lets autoregression do the rest. The prompt itself can be perfectly benign. By the time anything harmful is in flight, the prompt-time gate has already said yes and gone home.

This is not a tuning problem. The paper formalizes it: any defense that gates on a single layer’s activation alignment at prompt time admits crafted inputs that pass the gate while producing harmful continuations. You can move the gate, widen the cone, stack more prompts into the calibration set - the attack surface survives because the decision is made before the evidence exists.

In the measurements this shows up brutally. AlphaSteer, which zeroes out GCG and AutoDAN, sits at roughly 50% attack success against prefilling. The other prompt-time paradigms fare no better.

Reading the model after it starts talking

The asymmetry that saves us: the model may be coerced into complying, but its hidden states still encode what kind of turn it’s producing. That’s consistent with what I found in the sandbagging-detection work - internal representations carry intent that outputs alone hide.

So the defense moves to response time. Take the hidden states at the first few generated tokens, run a linear probe - the cheapest classifier there is - and ask: is this turn heading somewhere harmful? Across all seven models, that probe separates harmful from benign continuations at AUROC 0.97-1.00. Attach a halt rule and prefilling attack success drops to zero, with no false positives on benign traffic in my evaluation set.

Composition is where it gets satisfying. Response-time halting doesn’t replace prompt-time steering; it covers the region steering can’t see. Running AlphaSteer’s null-space steering for prompt-borne attacks plus the response-time halt yields defense success rates of 0.983 on Mistral and 0.994 on Llama across the full attack suite. Defense-in-depth, except the layers are actually orthogonal this time.

Here is the whole cross-product to play with - the defense-by-attack matrix from Table 1, per model. Select AlphaSteer and look at the prefilling row, then switch to the composed defense:

(Standalone and embeddable version: The blind-spot matrix.)

Why deployment people should care

Three practical notes from the receiving end of this result:

1. If your safety stack is prompt-time only, prefilling is an open door. This includes a lot of production setups where the “defense” is a classifier or steering pass over the user message. Anything with an assistant-prefill API surface, or agent frameworks that template partial responses, is exposed.
2. The fix is nearly free. A linear probe over first-token hidden states adds no meaningful latency and no extra model. If you already log activations for observability, you’re most of the way there.
3. Evaluate defenses on the cross-product, not the diagonal. A defense scored only against the attacks it was built for will look perfect right up until it isn’t. The per-sample results, attack implementations, and judge prompt are all released with the paper so the matrix is reproducible.

One research line, third leg

For regular readers, this slots into a single thread: discover diverse attacks automatically (RedQueen / MAP-Elites), track how vulnerabilities move across model generations (cross-generational transfer), and now close the defense gap those attacks expose. Attack discovery tells you where the walls are weak; this paper is about the part of the wall that wasn’t being watched at all.

Paper: Closing the Activation-Cone Blind Spot: Response-Time Probing and Unified Defense. Code, attacks, per-sample results, and the judge prompt ship with it. The defense-by-attack matrix is explorable as an instrument.

TL;DR: The claim that activation probes are too expensive to run in production is folklore. On compute it is wrong by six orders of magnitude: a linear probe is a dot product, roughly 0.00006 percent of the forward pass it rides on. Any real cost hides in implementation, in CUDA graph breaks, per-step device-to-host copies, and batching bookkeeping, none of which is intrinsic to probing. The systems community has already measured the adjacent problem, general model-internal observability, and the numbers back this up: naive hooks run an order of magnitude over baseline while a well-engineered asynchronous readout costs low single digits. What nobody has measured is the case that decides a safety architecture, an always-on calibrated probe library reported with operator-grade statistics. This is Part 1: the argument, what prior work already settles, the gap it leaves, and the harness aimed at that gap. Part 2 has the numbers.

Back in March, in the model honesty post, I wrote that I wouldn’t run probes on every request because the latency cost is real. It reads like engineering judgment. It was folklore. I had never measured it, so I went to find who had. The answer turned out to be more interesting than “nobody”: the systems community has already measured the adjacent problem, general model-internal observability, and those numbers puncture the folklore on their own. A naive forward hook runs an order of magnitude over baseline, and a readout that stays on the GPU and exports asynchronously costs low single digits. What is still missing is the measurement an operator actually needs before staking a safety case on it, and that gap is what this series is about.

It matters more than a benchmark footnote should, because this number quietly decides the architecture of every runtime monitoring stack being proposed right now. If activation probes are expensive, they get relegated to offline audits and sampled traffic, and the safety case for deployed agents rests on text-level signals alone. If they’re cheap, they belong in the serving path, on every token, always on. Those are two completely different systems, and once you step outside the observability papers, we’re still choosing between them on vibes.

So this is part 1 of settling it properly: the arithmetic, what the existing measurements already show, the specific gap they leave for a safety-probe library, and the design of a harness aimed straight at that gap.

The arithmetic says probes are free

Start with what a probe actually is at inference time. A linear probe is a dot product between a hidden state and a learned weight vector. That’s it. For a 7B-class model with hidden dimension 4096, one probe readout at one layer costs 4096 multiply-accumulates, call it 8,192 FLOPs per token.

The forward pass it’s riding on costs roughly 2 FLOPs per parameter per token, so about 14 GFLOPs per token for the 7B model. The ratio is 6 × 10⁻⁷. The probe is 0.00006 percent of the work you were already doing.

Scale it up to something aggressive: 100 probes at each of 10 layers, a thousand readouts per token. Now you’re at 8.2 MFLOPs per token, which is 0.06 percent of the forward pass. Go to a 70B model and the ratio gets better, because the forward pass grows faster than the probe does.

On compute alone, the “probes are too expensive” position is not just wrong, it’s wrong by six orders of magnitude. If the folklore is true, the cost has to be hiding somewhere else.

The same ratio, live. Pick a model and choose how many layers you read:

A systems engineer will object that I picked the flattering denominator, and they’d be half right. Decode is memory-bandwidth bound, not compute bound. You stream the entire weight matrix per token and do almost no arithmetic per byte, so FLOPs were never what set inter-token latency. Fine. Run the same argument in bytes: the probe weights for a thousand readouts are single-digit megabytes to move per token against the ~14 gigabytes you already move for the model, and the score buffer is rounding error. The ratio barely shifts. That the point survives the change of units is the tell. If a real cost exists, it is not in any quantity the arithmetic measures, FLOPs or bytes. It is in the launches, the copies, and the scheduling, and those never show up in a back-of-the-envelope at all.

Where the cost actually hides

And it might be. Modern serving engines are hostile environments for anything that wants to touch intermediate activations. Three suspects, in the order I’d rank them:

CUDA graph breaks. vLLM and friends capture the decode step as a CUDA graph and replay it. It’s one of the larger single wins in modern serving. A naive PyTorch forward hook is Python code executing in the middle of the forward pass, which is incompatible with graph capture, so attaching one can force the whole decode path back to eager mode. You’re not paying for the probe. You’re paying for losing graph replay on everything. This is not a hypothesis: an open vLLM internals plug-in restricts observation to the prefill phase precisely to preserve CUDA graphs, and pencils in a decode-time mode at roughly 25 percent throughput. That is the price of the naive path, already on the record.

Synchronization and device-to-host copies. A hidden state at d=4096 in bf16 is 8 KB per token per tapped layer. The bandwidth is nothing. But if your implementation copies scores to the CPU every decode step, you’ve inserted a synchronization point into a pipeline that lives or dies by staying asynchronous. Small syncs, repeated thousands of times per second, are how fast systems die.

Batching interactions. Continuous batching means the “batch” at any decode step is a shifting mix of requests at different positions. Per-request probe attribution has to slice a packed tensor correctly without adding its own bookkeeping overhead in Python.

Notice what all three have in common: none of them are intrinsic to probing. They’re artifacts of implementation strategy. Which suggests the real claim worth testing is not “probes are cheap” or “probes are expensive” but something more specific:

Probing is nearly free if the readout stays on the GPU and plays nice with graph capture. It’s expensive only when implemented the way a researcher would implement it in a notebook.

Here’s the shape of both paths:

Path A is what every interpretability codebase does today, because interpretability codebases were built for analysis, not serving. Path B packs all probes at a layer into a single weight matrix, does one small matmul per tapped layer inside the graph, accumulates scores in a device buffer, and exports asynchronously. The hypothesis is that Path B’s marginal cost is a few microseconds per step, which at typical inter-token latencies is measurement noise.

There’s a Path C too: fuse the readout into an existing kernel’s epilogue so it costs approximately nothing even in launch overhead. I’ve written enough Triton this year to believe that’s a weekend once Path B works. But I want to know whether it’s even necessary before writing it.

The experiment

The harness is deliberately boring, which is the point. Boring benchmarks are reproducible benchmarks.

The details that matter:

Four conditions on identical hardware and identical workloads: baseline vLLM with no probes attached, the naive hook implementation, the batched on-GPU implementation, and eventually the fused one. Each condition sweeps probe count (1, 10, 100 per layer) and tap depth (1, 4, 10 layers).

Metrics are the ones operators actually care about: time to first token, inter-token latency, and sustained throughput at a fixed request arrival rate, reported at p50, p95, and p99. Plus GPU memory delta and, critically, whether CUDA graph capture survived, because that single boolean probably explains most of whatever gap shows up.

Every number gets at least five runs, warmup discarded, and a bootstrap 95 percent confidence interval. If a difference isn’t distinguishable from run-to-run noise, it gets reported as indistinguishable, not rounded into a story. I built Spark-LLM-Eval (repo) because LLM evaluation without confidence intervals is astrology, and I’m not going to commit the same sin in a latency benchmark.

Predictions, written down before I have results

Two of these are, honestly, no longer wild bets. The observability results above already show naive hooks an order of magnitude over baseline and a well-engineered asynchronous readout in the low single digits. What none of them reports is the number for a calibrated safety-probe library run on every decode token, with the tail-latency statistics an operator would demand before trusting it. So these are falsifiable predictions for my specific configuration, not the general one, and I will be held to them.

1. The naive implementation (Path A) costs 15 to 40 percent of throughput on a 7B model, and almost all of it traces to the graph break and per-step synchronization rather than the probe math itself.
2. The batched on-GPU implementation (Path B) costs under 2 percent of throughput at 100 probes across 10 layers, and under 1 percent in the configurations anyone would actually run.
3. The fused version is statistically indistinguishable from baseline, making it a nice-to-have rather than a requirement.
4. Overhead as a fraction shrinks as model size grows, so the story gets more favorable exactly where the safety stakes are highest.

If prediction 2 holds for the safety-probe case, the last serious version of the objection dies with it. Not weakens. Dies. “We can’t afford to run probes in production” stops being a claim about the technique and becomes, at most, a claim about your implementation and your tolerance for a couple of percent.

What it unlocks

The reason to care about a percentage point of throughput is that it determines the monitoring policy you can honestly defend:

That ladder hides an assumption worth stating plainly: it is a monitoring ladder, not an enforcement one. If all you want is to log a score, decorate a trace, or raise an alert, the readout can leave the critical path entirely, and Path B’s async export costs you nothing. But the moment a probe is allowed to act, to block a tool call or halt a generation, you need its verdict before you commit the token, and “before you commit the token” is a synchronization point by definition. Enforcement puts the score back on the hot path that Path B worked to clear. So the honest ladder has two columns, not one: the overhead of always-on monitoring, which I expect to be trivial, and the overhead of always-on enforcement, which is a harder number and a different benchmark. Part 1 measures the first. The second is its own post, and I suspect its answer is exactly where selective probing earns its keep.

In the March post I sketched a three-layer honesty stack: confessions, chain-of-thought monitoring, and probes. The open question was always whether the probe layer could live in the serving path or had to stay in the lab. The observability community has been answering the general version of that question in public, wiring activation readouts into the serving engines themselves. Activation-level signals used to be the layer with the richest research results and the thinnest production tooling. That is changing fast. What has not been built is the part that matters for safety: not observability in general, but a calibrated, always-on probe layer measured to the standard you would hold a production safety control to.

None of this is hypothetical, and it is further along than I would have guessed. The systems community has built model-internal observability into serving engines and measured its cost: an asynchronous, GPU-resident readout in the low single digits of overhead, the naive hook path an order of magnitude worse, on real hardware with real workloads. The mechanism is settled and the folklore is dead. Two things are still open. Those results report medians for a debugging and observability use case, not the tail-latency statistics you need to run a safety control on every token. And none of them is a calibrated probe library wired to a monitoring policy. The runtime has been measured. The safety primitive that sits on top of it has not.

So that’s what I’m building, on top of the observability substrates the systems people just shipped rather than reinventing them. The harness here is the first milestone: standard tap points, a packed on-GPU readout, a calibrated probe library, latency reported to operator standards, and an adversarial eval asking whether a model that knows it’s being probed can evade detection. That last question is where this connects back to the Red Queen work, and it deserves its own post.

Part 2 will have the numbers, whichever way they land. If I’m wrong about prediction 2, that’s a more interesting post than if I’m right, and it redirects the design toward selective probing of high-stakes actions, which the March post already argued for. Either outcome is a measured number for the case that actually matters, which is the whole point.

If you’re running probes in production today, or you’ve measured any of this and I’ve missed your write-up, I want to hear from you: contact@subhadipmitra.com

References

1. Enabling Performant and Flexible Model-Internal Observability for LLM Inference (DMI-Lib). (2026). arXiv:2605.11093
2. vLLM Hook v0: A Plug-in for Programming Model Internals on vLLM. (2026). arXiv:2603.06588
3. Observation Plugin for Intercepting and Routing on Activations. vLLM RFC. vllm-project/vllm#36998
4. A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification. (2026). arXiv:2601.13288

The internet already has forty posts walking through the V4 model cards, so this won’t be another one. Instead, this post is organized around what you can actually do with DSpark’s two core ideas: how to tell if your serving stack has the problem they solve, whether the technique is worth your time at all, and how to train a drafter for your own model with the DeepSpec code DeepSeek released. My wrong assumption is just the doorway in.

Speculation is a scheduling problem, not a drafting problem

Quick recap of the mechanics. A small draft model proposes a block of tokens, the target model verifies the whole block in one forward pass, rejection sampling accepts the longest valid prefix plus one bonus token. Lossless by construction. Per-token latency is (T_draft + T_verify) / τ, where τ is tokens accepted per cycle.

On an idle GPU, verifying eight tokens costs about the same as verifying one, because decoding is memory bound and the extra tokens ride along in the batch. So you verify everything and it feels free. On a busy fleet it is not free: every speculative token you verify is compute taken from someone else’s request. Verify eight, accept two, and you’ve spent six tokens of a stranger’s latency on nothing.

DSpark’s response is a confidence head that scores each drafted token plus a scheduler that watches machine load. Quiet fleet: verify the full block. Slammed fleet: verify only the confident prefix, discard the tail unverified. Speculation becomes elastic, expanding into spare capacity and contracting out of the way. The shape of one decode cycle:

flowchart LR
    D["Drafter<br/><i>proposes k tokens</i>"] --> CH["Confidence head<br/><i>survival score per position</i>"]
    L["GPU load<br/><i>batch occupancy</i>"] --> S
    CH --> S{"Scheduler<br/><i>how much is worth verifying?</i>"}
    S -->|quiet fleet| VF["Verify all k"]
    S -->|busy fleet| VP["Verify confident prefix only"]
    VF --> T["Target forward pass<br/><i>one batch, one step</i>"]
    VP --> T
    T --> A["Longest accepted prefix<br/>+ 1 bonus token"]
    style S fill:#e76f51,color:#fff
    style A fill:#2d6a4f,color:#fff

What this means for you: if you run speculative decoding in a batched serving stack today, your acceptance rate is not the number to watch. Three numbers together tell you whether you have this problem: accepted length τ per verify cycle, batch occupancy at your p95, and the wasted-verify fraction, which is just (k − (τ − 1)) / k for draft length k. Your serving stack already exposes the raw material; vLLM, for instance, reports spec-decode acceptance counters you can derive the rest from. Plot wasted verify against occupancy, and if it climbs as the batch fills, you are paying the tax. Measure per-user latency at your p95 load, not at idle. If spec decoding looks great in your benchmarks and mediocre in production, wasted verification under load is the first suspect, and a fixed draft length is probably the cause. Even without adopting DSpark wholesale, making your speculation length load-aware (even crudely, like dropping draft length when batch occupancy crosses a threshold) captures part of the win.

Suffix decay, and why one token of context fixes most of it

Parallel drafters like Medusa and DFlash draft the whole block in one shot. Fast, but each position guesses independently, so accuracy rots deeper into the block. Position one is usually right, position five is a coin flip. This suffix decay is what caps τ. Sequential drafters like Eagle3 avoid it by drafting token by token, but they give back draft speed to get there, which is exactly the trade DSpark refuses to make.

DSpark keeps the parallel backbone and adds a tiny sequential head on top. Not an RNN over the whole prefix. A Markov head that conditions on exactly one token, the previous one, through a cheap low-rank projection. That alone holds acceptance steady deep into the block. The authors tried a full RNN head; it barely helped beyond the one-token version, so the one-token version ships.

Since DeepSpec ships all three drafter families side by side, the choice you’ll actually face is which one to train. Here is the trade in one table:

What this means for you: if you use a parallel drafter, suffix decay is your τ ceiling, and the fix is embarrassingly cheap. It also means longer draft blocks are back on the table: with the sequential head, stretching the block from 4 to 16 tokens costs almost nothing per round while meaningfully raising accepted length. If you tuned your draft length down to 3 or 4 because acceptance fell off a cliff, that tuning may be obsolete.

Should you use DSpark? An honest decision guide

You’re a good candidate if you control your model weights and your serving stack, you serve enough traffic that GPUs sit at meaningful utilization, and a real share of your workload is structured, meaning code, math, extraction, or tool calls. Structured outputs have fewer plausible next tokens, so accepted prefixes run long and the technique shines. Coding agents are the best case.

You’re a poor candidate if you rent inference through someone else’s API, because none of this is a switch you can flip from outside. You’re also a poor candidate on consumer hardware with a single user: speculative decoding needs the draft path to be roughly 10 to 30 times faster than the target, and with a weak speed ratio it can end up slower than plain decoding even though the outputs stay correct.

And one warning that doesn’t fit in a headline: drafters do not transfer. A draft head learns the output distribution of one specific target model. The checkpoint trained for stock V4 is the wrong drafter for your fine-tune, and wrong again for the same model in a different reasoning mode.

Can you run it today, without training anything?

Mostly, with early-adopter caveats. If your target model is stock V4, the released DSpark checkpoints load in vLLM and SGLang today; the community 1.5x number in the benchmarks section below came from exactly that path. What you’re getting is early support rather than a turnkey flag: formal DSpark spec-decode integration is still landing in both projects (there are open feature requests tracking it in SGLang and in vllm-ascend), so budget for version pinning and some config friction. Still, running a shipped checkpoint is a far lower bar than training a drafter, so try that first and only reach for DeepSpec when your weights diverge from stock.

How to train a DSpark drafter for your own model with DeepSpec

The genuinely useful part of this release is DeepSpec, the MIT-licensed codebase DeepSeek published for training and evaluating draft models. It ships implementations of DSpark, DFlash, and Eagle3, with released checkpoints targeting Qwen3 at 4B, 8B, and 14B plus Gemma-4-12B, and evaluation sets spanning math, code, and chat.

The pipeline, end to end:

flowchart LR
    P["Prompts<br/><i>from your real traffic</i>"] --> R["Regenerate answers<br/><i>with your target model</i>"]
    R --> C["Target cache<br/><i>~38 TB for Qwen3-4B</i>"]
    C --> TR["Train draft head<br/><i>target frozen, reuses its<br/>embedding and output layers</i>"]
    TR --> E{"Accepted length<br/><i>on held-out math, code, chat</i>"}
    E -->|holds up| DEP["Wire into serving stack"]
    E -->|falls short| P
    style C fill:#e76f51,color:#fff
    style DEP fill:#2d6a4f,color:#fff

The workflow is refreshingly honest about the drafter-target coupling, and it is concrete enough to walk through for real. Below is the actual run for the repo’s default target, Qwen3-4B. Everywhere Qwen/Qwen3-4B appears, your own fine-tune drops in; the whole point of the pipeline is that the drafter learns your model’s distribution, so there is no shortcut around regenerating the data yourself.

git clone https://github.com/deepseek-ai/DeepSpec.git
cd DeepSpec
python -m pip install -r requirements.txt

python scripts/data/download_and_split.py \
    --dataset-name mlabonne/open-perfectblend \
    --test-size 0.05 \
    --train-output-path train_datasets/perfectblend_train.jsonl \
    --test-output-dir eval_datasets \
    --skip-existing

bash scripts/data/launch_sglang_server.sh

python scripts/data/generate_train_data.py \
    --model Qwen/Qwen3-4B \
    --server-address \
        127.0.0.1:30000 127.0.0.1:30001 127.0.0.1:30002 127.0.0.1:30003 \
        127.0.0.1:30004 127.0.0.1:30005 127.0.0.1:30006 127.0.0.1:30007 \
    --concurrency 32 \
    --temperature 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
    --max-tokens 4096 \
    --disable-thinking \
    --resume \
    --input-file-path train_datasets/perfectblend_train.jsonl \
    --output-file-path train_datasets/qwen3_4b/perfectblend_train_regen.jsonl

python scripts/data/prepare_target_cache.py \
    --config config/dspark/dspark_qwen3_4b.py \
    --train-data-path train_datasets/qwen3_4b/perfectblend_train_regen.jsonl \
    --output-dir ${HOME}/.cache/deepspec/qwen3_4b_target_cache \
    --local-batch-size 16

bash scripts/train/train.sh

bash scripts/eval/eval.sh

Two sanity checks worth doing once the eval runs. Run the released DSpark checkpoint for your base model through the same eval to calibrate what good looks like. And compare your accepted length on the code sets against the chat sets: if code is not clearly ahead, your training data does not match your traffic, and the problem is in steps 1 and 2, not in training.

Budget warning, restated as a summary: the expensive step is the data, not the training. If you control your weights, this is still a weekend-to-a-week project rather than a research program, but it is not a laptop project.

DSpark benchmarks, for the record

I’ve argued the numbers aren’t the story, but you should have them in one place. Offline, DSpark’s accepted length beats Eagle3 by roughly 27 to 31 percent and DFlash by 16 to 18 percent across the three Qwen3 sizes, and a 2-layer DSpark drafter outperforms a 5-layer DFlash. Growing the draft block from 4 to 16 tokens costs only 0.2 to 1.3 percent extra per-round latency. In production, against DeepSeek’s previous single-token MTP-1 baseline at matched capacity, per-user generation runs 60 to 85 percent faster on V4-Flash and 57 to 78 percent faster on V4-Pro, with aggregate throughput up 51 to 52 percent at fixed per-user speed targets. The shipped config is DSpark-5, five-token blocks with the Markov head.

Two calibration notes. The roughly 400 and 660 percent figures you’ll see elsewhere (406 percent on V4-Pro at a 50 token-per-second per-user target, 661 percent on V4-Flash at 120) are real but describe throughput at speed targets strict enough that the single-token baseline nearly collapses, so the ratio balloons. They mark the corner of the serving frontier, not the typical case. And the baseline matters: an early community vLLM benchmark on V4-Flash landed around 1.5x over MTP-1 and 2.3x over plain decoding, which is the kind of number you should expect to reproduce, not the headline.

Build intuition before you build anything

Before you spend a week on DeepSpec, spend a minute on the simulator below. It runs three decoding policies against the same GPU budget: plain decoding, fixed-length speculation, and DSpark’s confidence-scheduled version.

Here is the experiment to run. Set the workload to code, then drag concurrent requests up and watch the frontier chart. At some point the fixed-length speculation curve crosses below plain decoding. That crossover is the entire argument of this post in one pixel: past saturation, a fixed draft length keeps paying to verify tokens that were never going to be accepted, and speculation becomes a tax on every other request. DSpark’s curve doesn’t cross, because the scheduler shrinks its verify window as utilization climbs. You can watch it happen in the lanes above the chart, where the scheduled policy starts tearing the tail off its blocks while the fixed policy keeps burning red.

Then switch the workload to chat and see how much earlier everything degrades when acceptance decays fast, and toggle the Markov head to see why one token of context moves the whole frontier. The sliders map directly onto your deployment questions: how accurate is your drafter, how long can your blocks be, and how busy are your machines.

(Standalone and embeddable version: Speculation under load.)

The constants in the toy are illustrative, not fitted to the paper. The dynamics are the real thing.

Sources

• The DSpark paper: github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf
• DeepSpec codebase: github.com/deepseek-ai/DeepSpec
• DSpark checkpoints for V4-Pro and V4-Flash on Hugging Face
• My earlier post on building speculative decoding from scratch: Making LLMs faster
• The simulator as a standalone instrument: Speculation under load

TL;DR: The June 2026 discourse on loop engineering is right about the shift and wrong about the hard part. A loop always converges. That is what it is built to do. But convergence is not correctness: a loop keeps whatever passes your check, so it settles on the cheapest output that clears the bar you actually wrote, not the one you meant. Every gap between “passed the check” and “actually correct” gets found and exploited, by accident or by an adversary, and the check that separates the two is the part that decides whether the loop compounds your work or builds an expensive machine for being confidently incorrect at scale. This is the practitioner’s playbook, written from running loops unattended over real production systems and not just a coding agent in a terminal.

Everyone is getting the shift right. Most are skipping the hard part.

Within two weeks in June 2026, three people said the same thing and a term went viral.

Peter Steinberger: “You shouldn’t be prompting coding agents anymore. You should be designing loops that prompt your agents.” Boris Cherny, who runs Claude Code at Anthropic: “I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.” And Addy Osmani gave it a name: “Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead.”

They are right. The default unit of work is moving from the response to the trajectory. The bottleneck is no longer “can the model do this in one shot” but “can I design a system that drives the model to a correct end state on its own.” That is a real change, and if your mental model is still copy, paste, re-prompt, you are about to be out-leveraged by people who automated the re-prompting.

But read the wave of explainers that followed and you will notice something. They all describe the loop. Reason, act, observe, repeat. They draw the circle. Then they move on. The circle is the part that is genuinely easy. A while loop with a tool-calling model inside it is a weekend project. I have watched teams build that circle in an afternoon and then spend three months discovering that the circle is not the problem.

The problem is the one box almost nobody dwells on: verify. The box that decides whether the loop goes around again or stops. That box is where loops create value or destroy it, and it is the part of loop engineering that is actually hard, actually technical, and actually where your judgment as an engineer goes.

This post is about that box.

What loop engineering actually is, stated precisely

The terms are getting muddy as the content farms amplify them, so be precise. The lineage is four layers, each wrapping the last:

• Prompt engineering optimizes a single instruction you type.
• Context engineering optimizes what the model sees on a given call.
• Harness engineering optimizes the environment one agent run executes in. An agent equals a model plus a harness, where the harness is everything that is not the model: tools, memory, guardrails, orchestration, context, sandbox, serving. In production it decides whether your agent works far more than the model choice does.
• Loop engineering optimizes the controller that runs agents against a goal until the goal is met or an exit fires.

A harness equips one run; a loop decides whether to run again. You can have an excellent harness and a terrible loop, and the result is a capable agent that marches confidently off a cliff because nothing above it is checking whether it is still pointed at the goal. The harness is the agent. The loop is the supervision, and this post lives in the supervision layer, because that is where the check lives.

Here is the canonical loop, with the box this post is about marked in orange and the exits marked explicitly, because a loop without exits is not an architecture, it is an outage waiting for a credit card.

flowchart LR
    I["Objective<br/><i>a verifiable goal</i>"] --> C["Context<br/><i>assemble state</i>"]
    C --> A["Act<br/><i>tools, edits, calls</i>"]
    A --> O["Observe<br/><i>run it, capture results</i>"]
    O --> V{"Verify<br/><i>did we meet the goal?</i>"}
    V -->|pass| DONE["Exit: done"]
    V -->|fail| ADJ["Adjust<br/><i>update the plan</i>"]
    ADJ --> C
    V -.->|cap hit, budget out, no progress| STOP["Exit: stop and escalate"]
    style V fill:#e76f51,color:#fff
    style DONE fill:#2d6a4f,color:#fff
    style STOP fill:#9d0208,color:#fff

Notice that everything interesting routes through Verify. The quality of your loop is bounded by the quality of that diamond. Everything else is plumbing.

The independent claim: convergence is not correctness

Here is the thesis the explainers are missing, and it is not a vibe, it is a mechanism. A loop always converges. Reaching a stable “done” is the one thing a loop is built to do reliably. The trap is reading that convergence as correctness, because the two are not the same event, and a loop is a machine for driving them apart.

A loop fails in two directions. It can fail to converge at all, thrashing or running away until something stops it, which is loud and which the exits later in this post are for. Or it can converge cleanly on the wrong thing, which is silent, which no exit catches, and which is the failure this post is mostly about.

When you prompt a model once, you are exposed to its error distribution exactly once. It is right or wrong, you look at the result, you move on. Your judgment is in the loop on every output.

When you wrap that model in a loop, you change the statistics completely. The loop generates a candidate, runs your verifier, and keeps the candidate only if the verifier passes. Run that ten or fifty times and you are no longer sampling the model’s outputs. You are selecting for outputs that pass your verifier. That is, by definition, optimization. Your verifier is the objective function, and the loop is a crude optimizer hammering on it. So the loop converges on exactly one thing, the cheapest output that satisfies the check, and whether that is also correct depends entirely on how tightly the check pins passing to being right. Nothing in the loop is working to close that gap. The loop is working to exploit it.

This is Goodhart’s law with a motor on it. When a measure becomes a target it stops being a good measure, and a loop turns your check into a target and then attacks it hundreds of times. Any gap between “passes the check” and “is actually what I wanted” does not stay a harmless gap. It becomes a reliable exploit, found by brute force, because the loop keeps trying until something makes the light go green.

I have watched this happen in the most boring possible ways. A loop told to “make the tests pass” deletes the failing test. A loop told to “reduce lint errors” adds suppression comments. A loop graded by an LLM judge learns to write outputs that sound authoritative to the judge, because confident prose is cheaper to generate than correct work. Each one converged. None of them was correct. They are the natural consequence of pointing an optimizer at a proxy.

This reframes the entire critique conversation. The viral skepticism around “loopmaxxing,” the idea that infinite iterations will fix anything, is correct but underspecified. Loopmaxxing is not a discipline problem. It is a convergence-without-correctness problem. More iterations do not help because you are climbing a hill whose summit is “satisfies a flawed check,” not “is correct.” More compute just gets you to the wrong summit faster.

This is the same failure surface as reward hacking in reinforcement learning, and it is why the RLVR crowd became obsessed with verifiable rewards. It is also exactly the territory I spend my research time in. A model that has learned what a grader rewards, rather than what the task requires, is doing a form of sandbagging against the evaluator, and a loop is a machine for finding that behavior whether you wanted it or not. If you take one thing from this post: the loop does not make your agent smarter, it makes the check the only thing standing between “it stopped” and “it is right.”

You can watch this play out. Weaken the verifier or turn up the gaming and correctness collapses while the loop still converges to a confident “done”:

The verifier is your specification, made executable

Everything now rests on that check, so it is worth being precise about what it actually is, because that changes how you write it. The verifier is not a test you bolt onto the work after the fact. It is the specification of the work, written in the one language an optimizer respects: a check it cannot talk its way around.

The loop never reads your intent. It cannot. It reads the verifier, over and over, and shapes its output toward whatever the verifier will accept, so the verifier is the complete, operative definition of done inside your system. Anything you meant but did not encode does not exist. The spec in your head, the ticket, the design doc, none of it is binding, because none of it is what the loop can see.

This is why “I will know it when I see it” is fatal in a loop in a way it never was by hand. When you were in the loop on every output, your taste was the spec, applied live. Step out and let it run unattended, and your taste has to be written down or it is gone. A vague verifier is not a lenient spec, it is the absence of one, and the loop fills that vacuum with the cheapest output that clears the bar you wrote, not the bar you meant.

So the discipline is blunt: if you cannot write the verifier, you have not finished specifying the task. That is not a tooling gap you route around with a smarter model. It is the task telling you that you do not yet know what you want. Writing the verifier first is the act of finishing the specification, and most of the time it is where you find the spec was never finished at all.

The verifier ladder

So the work of loop engineering is mostly verifier engineering. The single most useful mental model I have for this is a ladder, ordered by how cheap, deterministic, and ungameable each kind of check is. The rule is simple and it is the highest-leverage habit in this whole discipline: push every check as far up the ladder as it will go.

Most teams start at the bottom because it is the easiest to wire up. “Have a second model grade the output” is one API call. It is also the most gameable verifier on the ladder, and you are about to point an optimizer at it. That is precisely backwards.

A few practical moves that pay off immediately:

• Convert “looks right” into “passes a check.” If your stop condition is “the refactor is clean,” you have no verifier, you have a wish. Replace it with: the test suite still passes, public API signatures are unchanged, and the diff touches no file outside the target module. Those are deterministic. The loop cannot argue with them.
• When you have no ground truth, manufacture a proxy for it. Differential testing (run the old and new code on the same inputs, outputs must match), metamorphic testing (a known input transformation must produce a known output transformation), and golden datasets give you deterministic checks for tasks that look unverifiable at first glance. Most “you cannot test this” claims are a failure of imagination about oracles.
• If you must use an LLM judge, anchor it and make it adversarial. Give it the spec, the artifact, and a rubric, and instruct it to find the strongest reason to reject. A judge prompted to approve approves. A judge prompted to refute, with the original objective in hand, is a meaningfully harder target. I built Spark-LLM-Eval partly because evaluating model output rigorously, with confidence intervals and significance rather than a single judge call, is exactly the muscle loop engineering demands and most teams have not built.

Make the checker independent, or it shares the maker’s blind spots

Osmani’s primitives for a real loop include a maker and a separate checker, and Cobus Greyling makes the same point: a separate model verifies, so the agent never grades its own work. This is correct and it is incomplete. The detail that matters is what the checker is allowed to share with the maker.

If your maker is model X with context C, and your checker is also model X with context C, you have not built independent verification. You have built an echo. The two share an architecture, a training distribution, and a working memory, which means they share blind spots. Whatever the maker confidently got wrong, the checker will confidently bless. The loop then converges, smoothly and efficiently, on output that is wrong in a way both halves cannot see.

Independence is a spectrum and you want as much of it as you can afford. In rough order of strength: a deterministic check shares nothing with the maker and is the strongest checker you can have. A different model family with a fresh context is next. The same model with a fresh context and only the spec plus the artifact, not the maker’s reasoning, is weak but better than nothing. The same model continuing the same conversation is not a checker at all.

The concrete rule I follow: the agent that decides “done” must not be the agent that did the work, and it should see objective signals before it sees prose. Hand it the test results, the diff, and the schema validation first. Let it read the maker’s explanation last, if at all. Explanations are where models launder their own mistakes into something that sounds verified.

Who verifies the verifier?

Here is the question this entire conversation walks up to and then declines to ask. You have made the verifier the thing every other claim in this post depends on. You have climbed the ladder and made the checker independent. So how do you know the verifier itself is trustworthy? A load-bearing beam you never tested is not a feature. It is the subject of the eventual incident report.

A verifier is software, and the unsettling part is that it is the one piece of software in your stack with nothing checking its work. The maker is checked by the verifier. The verifier is checked by nothing, unless you build that too. Three habits close the gap.

Mutation-test the verifier. Try to fool it on purpose. Borrow the move from mutation testing: take outputs you already know are wrong and confirm the verifier rejects every one of them. Delete the function the tests cover and watch the suite go red. Hand the reconciliation checker a set of matches that secretly net to a non-zero figure and confirm it refuses to pass. Feed the LLM judge a confident, well-formatted, factually wrong answer, which is exactly what a loop will eventually generate, and see whether it catches the error or gets flattered by the formatting. A verifier you have never tried to defeat is one you are trusting on faith, right before you point an optimizer at it that will try to defeat it for a living. Do that job first, in daylight, before the loop does it at three in the morning. The checks that survive are the ones you can let run unattended; the ones that fold are gaps the loop would have found anyway, and you found them cheaper.

Measure the gap. Do not assume it is closed. Your entire exposure is the distance between passed the verifier and actually correct, so treat it as a number you watch, not something you reasoned away. Sample the outputs that passed, audit them against a stronger, slower oracle, and the rate at which a pass turns out wrong is your verifier’s false-acceptance rate. It is the most honest single measure of your loop’s quality, and it belongs on a dashboard next to latency. If it reads zero, your sample is too small or your audit too shallow. When it drifts upward, your verifier is decaying or your inputs have moved out from under it, and you want that from a graph, not from a customer.

Assume the verifier rots. It is code, and code rots, but a verifier rots silently, which is worse. The golden dataset ages out of its distribution. The spec it encodes drifts as the product changes. The threshold that was right last quarter is loose this quarter. And because a stale verifier still returns a crisp, confident pass, the loop keeps shipping and the gap widens with nobody watching. So treat the verifier as a maintained asset with an owner and a review cadence: re-derive the golden set on recent data, re-run the mutation tests after every meaningful spec change, and watch the false-acceptance number for the slow creep that means the check has quietly stopped meaning what it used to.

Let the model write the verifier. Never let it be the verifier.

There is a tempting shortcut and a real technique here, and they look almost identical. The shortcut is to let the model be the verifier: ask it, every iteration, whether the output is good. You know how that ends, because the loop optimizes the model into saying yes. The technique is its mirror image: let the model write the verifier, once, offline, then take the model out of the decision entirely.

Models are genuinely good at proposing oracles. Ask one to enumerate the invariants a correct reconciliation must satisfy, or to generate property tests and metamorphic relations for a function, or to brainstorm the adversarial inputs a check ought to survive, and it will hand you a strong, broad list, often broader than you would have written yourself. That is the model doing exactly what it is good at: generating candidates over a space you then prune. What you do next is the whole trick. You read those candidate checks, you keep the ones that are actually right, you freeze them into deterministic code, and you delete the model from the loop’s verify step. The model was a test author at build time. It is not the test at run time.

The line is build time against run time, and it maps onto the ladder: drafting a check is a stochastic judgment, but freezing it into code carries it up to a deterministic one an optimizer cannot argue with, at a cost of one call instead of one per iteration forever. The best verifiers I have are often drafted by a model. None of the good ones are still run by one.

The exits are non-negotiable

A verifier tells you when to stop because you succeeded. You also need to stop when you are failing, and that is a different set of mechanisms. Robust loops carry several independent exits, and “independent” is load-bearing: if your only way out is the verifier passing, a loop that can never pass will run until something else breaks, usually your budget.

No-progress detection is the one people skip and then regret. A loop without it does not fail loudly, it fails by spending three hours and forty dollars oscillating between two almost-identical states, each of which fails the verifier in the same way. The cheap version: hash the candidate diff each iteration and stop if you see the same hash twice. The better version: track the verifier’s score over iterations and stop when the marginal improvement per iteration falls below a threshold. You are watching for a derivative going to zero, which is a control-theory instinct, not an AI one, and that is the point.

Here is the shape of a loop that will not bankrupt you, in pseudocode, with every exit wired in:

def run_loop(objective, budget, max_iters=12):
    state = gather_context(objective)
    seen = set()
    last_score = 0.0
    for i in range(max_iters):
        if budget.exceeded():                    # tokens, dollars, wall-clock
            return escalate("budget", state)
        artifact = maker.act(state)              # the model does work
        signals = observe(artifact)              # run it, capture deterministic results
        result = checker.verify(objective,       # INDEPENDENT checker,
                                artifact,        # deterministic signals first,
                                signals)         # prose last
        if result.passed:
            return done(artifact)
        fingerprint = hash_state(artifact, signals)
        if fingerprint in seen:                  # oscillation / no progress
            return escalate("stalled", state)
        if result.score - last_score < EPSILON:  # diminishing returns
            return escalate("plateau", state)
        seen.add(fingerprint)
        last_score = result.score
        state = adjust(state, result.feedback)
    return escalate("max_iters", state)

The model appears on exactly one line. Everything else is the loop, and the loop is where your engineering lives now.

What changes at enterprise scale

Everything above holds for a coding agent in your terminal. I want to spend the rest of this on the part the current discourse barely touches, because it is where I have spent the last few years and it is where loop engineering gets genuinely hard: loops that run unattended, over real systems, inside an organization.

When the loop is a coding agent, you have a gift you may not appreciate: a free, deterministic, fast verifier sitting at the top of the ladder. The test suite. The compiler. The world has handed you ground truth.

Now take the gift away. I built a system of more than fifteen agents to consolidate a sprawling, fragmented BigQuery estate into a single coherent foundation layer, and a separate multi-agent system that migrated tens of thousands of notebooks across thousands of projects, cutting each migration from roughly a week to under thirty minutes. There is no pytest for “did we consolidate this schema correctly” or “is this migrated notebook semantically equivalent to the original.” The verifier you get for free in software does not exist. You have to build it, and building it is most of the actual work.

What you build instead of a test suite is a verification coalition: several independent, mostly deterministic checks that together bound the space of “wrong” tightly enough to trust the loop.

flowchart TB
    T["Trigger<br/><i>schedule, event, queue</i>"] --> CTX["Context assembly<br/><i>skills, durable state, retrieval</i>"]
    CTX --> MK["Maker agents<br/><i>worktree-isolated, parallel</i>"]
    MK --> VC["Verification coalition"]
    VC --> D1["Schema and contract checks"]
    VC --> D2["Data diff vs golden sample"]
    VC --> D3["Policy and rule engine"]
    VC --> D4["Sampled human review"]
    D1 & D2 & D3 & D4 --> G{"Gate"}
    G -->|pass, low blast radius| APPLY["Apply and record state"]
    G -->|high blast radius| HUMAN["Human approval"]
    G -->|fail| ADJ["Adjust and retry"]
    ADJ --> CTX
    BUD["Budget governor<br/><i>tokens, dollars, wall-clock, iterations</i>"] -. enforces .-> G
    style VC fill:#264653,color:#fff
    style BUD fill:#e9c46a,color:#1a1a1a
    style HUMAN fill:#e76f51,color:#fff
    style APPLY fill:#2d6a4f,color:#fff

Three things change once you are operating at this altitude, and none of them are in the viral threads.

Ground truth is a budget item. For the notebook migration, the strongest verifier is “run the original and the migrated notebook on the same data and diff the outputs.” That is differential testing, it sits high on the ladder, and it costs real compute per notebook. So verification is not free and it is not unlimited, which means you triage: deterministic cheap checks on everything, expensive differential checks on a risk-weighted sample, human review on the long tail the cheap checks flag. The verifier coalition is itself a cost-optimization problem. Most people think the model calls are the expense. At scale, verification is frequently the larger line item, and it should be, because it is the part you can trust.

Budgets stop being a safety rail and become the governance layer. When one engineer runs one loop, a budget is a guardrail against a runaway. When an organization runs hundreds of loops, the budget allocation is the control plane. Which loops get to touch production. How much they can spend before a human signs off. What blast radius is allowed without approval. When I set up an operating model for agents across a large delivery organization, the hard part was never the agents. It was making agentic checks a mandatory gate, standardizing what a sufficient verifier looks like, and giving budgets and human gates teeth so that “we automated it” never quietly meant “we stopped checking it.” That is loop engineering as an organizational discipline, and it looks a lot more like SRE error budgets than like prompt design.

You debug trajectories, not outputs. Once the unit of value is the trajectory, the unit of debugging is too. When a loop produces a bad result on notebook 18,000 of 30,000, “look at the output” is useless. You need the full trace: every step, every tool call, every verifier decision, the cost attributed per step, and the ability to replay it. This is observability work, and it is closer to distributed-systems debugging than to anything in the prompt-engineering playbook. It is also where interpretability earns its keep in production. Being able to look inside a step and ask why the model did what it did, which is the circuit-tracing work I have pushed into production settings, turns a black-box trajectory into something you can actually fix.

A worked example: a reconciliation loop, in banking and in ADK

Abstractions are cheap, so here is a concrete one from a domain I have spent years in. Intersystem reconciliation: a bank’s ledger has to agree with a counterparty feed, a payment processor’s file, or a sub-ledger, across millions of transactions a day. The breaks, the mismatches, have to be found, classified, explained, and resolved, and any adjustment above a materiality threshold needs a human sign-off. It is high-volume, repetitive, judgment-laden in the middle and rigid at the edges. It is exactly the kind of work people now want to wrap in a loop.

It is also a good loop, and the reason is the whole thesis of this post: reconciliation comes with a real verifier. The books either balance or they do not. That is a deterministic check sitting at the very top of the ladder, which means you can let an agent iterate without flying blind.

• The verification coalition
• The same loop in ADK

• Here is how the verification coalition lays out, mapped to the ladder:

  • Top of the ladder, deterministic, run on everything. Control totals reconcile to source-system balances. Every proposed set of matches nets to zero. Every transaction appears exactly once. Reason codes come from a closed vocabulary, not free text. These are not model calls. They are arithmetic and set operations, and they cannot be flattered.
  • Middle, rule engine. Break classification (timing, fee, FX, duplicate, missing), plus policy checks: materiality thresholds, segregation of duties, cutoff rules.
  • Bottom, LLM judge, used sparingly. The only place a model gets to grade anything is the human-readable narrative of a break, and even there it is anchored to the deterministic classification. The model never decides whether a break is real or resolved. That decision lives in the arithmetic.
  • Human gate. Any adjustment or write-off above the materiality threshold stops and waits for an approver. High blast radius, non-negotiable.

• Now the part the framework gives you, and the part it does not. Google’s Agent Development Kit ships a LoopAgent that runs sub-agents on a loop with a hard iteration cap, and any sub-agent or tool can break the loop by setting tool_context.actions.escalate = True. So the skeleton is almost trivial:

  from google.adk.agents import LoopAgent, LlmAgent
  from google.adk.tools import FunctionTool, ToolContext
  
  def run_verifier(tool_context: ToolContext) -> dict:
      # The verification coalition. Deterministic checks first, no model in sight.
      breaks  = balance_and_netting_checks(tool_context.state)  # nets to zero, control totals, no dupes
      breaks += policy_rule_checks(tool_context.state)          # materiality, segregation of duties, cutoff
      if not breaks:
          tool_context.actions.escalate = True                 # verifier passed -> exit the loop
          return {"status": "reconciled"}
      return {"status": "open", "breaks": breaks}              # feed the breaks back for another pass
  
  matcher = LlmAgent(
      name="matcher",
      instruction="Propose matches and classify the open breaks. Never declare anything resolved.",
      tools=[...],  # ledger lookups, fuzzy matching, FX conversion
  )
  verifier = LlmAgent(
      name="verifier",
      instruction="Call run_verifier. You never decide a break is resolved yourself; the checks do.",
      tools=[FunctionTool(run_verifier)],
  )
  
  recon_loop = LoopAgent(
      name="reconciliation_loop",
      sub_agents=[matcher, verifier],  # maker, then independent checker
      max_iterations=6,                # a hard exit that does not depend on success
  )
  

  Look at what ADK actually handed you: the loop, the iteration cap, and the escalate hook. That is the easy twenty percent. What it did not hand you is balance_and_netting_checks, policy_rule_checks, and the rule that a model never gets to call a break resolved. That deterministic coalition, and the discipline of keeping the model out of the verify decision, is the other eighty percent, and it is the part that decides whether this runs unattended against the general ledger or quietly books a wrong adjustment at three in the morning.

  Two details this makes concrete. First, ADK gives you max_iterations and escalate, but the budget, wall-clock, and no-progress exits from the table above are still yours to wire, through callbacks or the runner. The framework gives you one exit cleanly and expects you to build the rest. Second, the contrast with a bad loop is sharp. Point the same LoopAgent at a goal like “improve this customer’s credit assessment” with an LLM judge as the only verifier, and you have built a machine that optimizes for assessments that please a grader. One of these belongs in production. The other is a model-risk and compliance incident waiting for a regulator. The framework cannot tell the difference. Only the verifier can, which is the entire point.

The verifier is also your security boundary

Everything so far has assumed the gap between passes the check and is correct gets found by accident, the loop stumbling onto the cheapest output that clears the bar. Now assume someone is steering. The moment a loop runs over input that an adversary can influence, the verifier stops being only a quality gate and becomes a security boundary, and almost nobody in this discourse is treating it as one.

The attack surface is wider than it looks, because reading untrusted content is part of the loop’s normal operation, not an edge case. A coding agent runs over a repository whose README or dependency can carry an injected instruction. A reconciliation loop ingests a counterparty feed a motivated party has every reason to shape. A research loop reads web pages written specifically to be read by an agent like this one. Prompt injection here is an attempt to turn your maker against your own objective from inside the data it was told to trust, and the maker, being helpful, tends to comply.

What stands between that and a bad write is the verifier, and the adversarial framing sharpens which verifiers you can tolerate. A deterministic check high on the ladder is harder to suborn, because arithmetic does not read instructions: the books either balance or they do not, regardless of what the counterparty feed whispered to the maker on the way in. An LLM judge, by contrast, is reachable by the very same injection that compromised the maker, through the very same channel. So independence gains a second meaning: independence from the maker’s blind spots was the quality argument, and independence from the maker’s compromise is the security one, because a checker that shares the maker’s context shares whatever just hijacked it.

The convenient part is that the discipline does not change, so you do not need a separate security story bolted on afterward. Climb the ladder and the injection has less to grab onto. Keep the model out of the verify decision and a compromised maker cannot launder its output past the gate. Put a human on the high blast radius actions and the worst an injection buys is a proposal, not a posting. The verifier you built to stop the loop from fooling itself is, with no extra work, the one that stops someone else from fooling it on purpose.

Loopmaxxing and the failure modes, mapped to their real cause

The failure modes getting passed around are real. What is missing is that almost all of them are the verifier failing in disguise. Here is the map I use.

Two of these deserve a reflexive habit.

Distill and demote. Once a step is stable, replace the model call with code. An agent that has classified the same input the same way a thousand times is not a stochastic job anymore, it is a function. The mature loop makes fewer model calls over time, not more, because you keep promoting proven behavior into deterministic code, which also moves checks up the ladder. The best loop engineering deletes LLM calls.

Comprehension debt is paid later. Osmani’s warning is the right one to close on: “No volume of recursive loop cycles can salvage a poorly specified objective or unprincipled software architecture.” A loop ships faster than you can understand what it shipped, and that gap is debt against a future incident at a brutal interest rate.

It is not new, and respecting that makes you better at it

This was not invented in June 2026. ReAct, Reflexion, evaluator-optimizer and generator-critic patterns, RLVR: the loop is old, and underneath all of them is the feedback control loop that control theory formalized a century ago and SRE has run production on for two decades. A control loop always converges, but only to whatever its sensor measures; if the sensor is wrong, it converges confidently to the wrong place. We renamed the sensor “verifier” and the actuator “tool call,” and the math did not change. What is new is only the precondition: the model at the center is finally good enough that running it autonomously to a goal beats driving it by hand, so the bottleneck moves outward, from the model to the loop. “It is just a while loop” is correct about the syntax and wrong about the work, the same way “a database is just a file” is correct and useless.

The playbook

If you are building your first serious loop, do these in order. The order is the advice.

1. Write the verifier first. If you cannot state a check that separates done from not-done without the word “good,” you are not ready to build the loop. You are ready to specify the objective better.
2. Climb the ladder. For every check, ask whether it can be made more deterministic and moved up. Spend your effort here, not on the prompt.
3. Make the checker independent. Different model or fresh context, deterministic signals before prose, never the maker grading itself.
4. Test the verifier before you trust it. Mutation-test it: feed it outputs you know are wrong and confirm it rejects them. The check you never tried to fool is what the incident report is about.
5. Wire every exit before the first unattended run. Iteration cap, token and dollar budget, wall-clock, no-progress detector, human gate. No exits, no sleeping.
6. Budget per trajectory and alert on it. Cost is a property of the trajectory now. Treat a cost spike like a latency spike.
7. Instrument the trajectory. Full traces, replay, per-step cost. You cannot debug what you cannot see, and you will have to.
8. Track the verifier gap. Sample what passed, audit it against a stronger oracle, watch the false-acceptance rate. A rising gap is your verifier rotting in public.
9. Distill and demote. Each week, find the step that has earned its way into plain code, and promote it.
10. Keep a human on the high-blast-radius gate. Automate the verification. Do not automate away the accountability.

Honest assessment: when to reach for a loop

• Reach for a loop when
• Do not reach for a loop when

• • The task is multi-step and benefits from real feedback between steps.
  • You can express “done” as a check that sits high on the verifier ladder.
  • The cost of an iteration is much smaller than the value of getting it right unattended.
  • You can afford to build the verifier and the observability, not just the agent.
• • Your only available verifier is “an LLM thinks it looks good.” That is not a verifier, it is a target, and you are about to optimize against it.
  • The goal is genuinely subjective with no proxy for ground truth. Loops do not create signal that is not there.
  • A single well-engineered call already solves it. Not everything needs to be autonomous, and a loop around a solved problem is just latency and cost.
  • You cannot afford the verification and observability the loop demands. A loop you cannot watch is a liability, not a productivity gain.

Loop engineering is a real discipline and the shift it names is real. But the marketing has the emphasis exactly backwards. The loop is not the achievement; any loop converges. Converging on something correct is the achievement, and that is a verifier you can trust an optimizer to attack a thousand times, and at enterprise scale, a coalition of them, governed by budgets, instrumented end to end, with a human on the gate that matters.

Osmani’s closing line is the right one to steal: “Build the loop. But build it like someone who intends to stay the engineer, not just the person who presses go.” The way you stay the engineer is by owning the verify box. Everything else, the model will increasingly handle. That box is yours.

Building loops over real systems and wrestling with the verifier problem? I would like to compare notes: contact@subhadipmitra.com

References

1. Osmani, A. (2026). Loop Engineering. addyosmani.com/blog/loop-engineering (June 7, 2026)
2. Steinberger, P. (2026). Remarks on designing loops that prompt agents, via X (June 7, 2026).
3. Cherny, B. (2026). Remarks on loops at @Scale, via TechTalks (June 21, 2026).
4. Dickson, B. (2026). Demystifying loop engineering: get more from AI agents, avoid loopmaxxing. TechTalks. bdtechtalks.com (June 22, 2026)
5. Greyling, C. (2026). Loop Engineering. cobusgreyling.substack.com (June 9, 2026)
6. decodingAI (2026). Agentic Harness Engineering: LLMs as the New OS. decodingai.com (March 31, 2026)
7. Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629
8. Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366
9. Google. Agent Development Kit (ADK): Loop agents. google.github.io/adk-docs

TL;DR: OpenAI is showing ads in ChatGPT. Perplexity tried and pulled back. Google is taking a measured approach. Meanwhile, the real action is happening underneath: regions of vector embedding space near high-value queries are becoming the new commercially contested territory - the “shelf space” of the AI era. GEO (Generative Engine Optimization) and RAG poisoning are points on the same spectrum, and nobody is connecting the security research, the marketing industry, and the mechanism design papers. This post maps the landscape, identifies the gaps, and proposes a framework for thinking about embedding space as an economic system.

Three months ago, OpenAI flipped a switch and started showing ads inside ChatGPT. Criteo signed on as the first ad-tech partner. Smartly followed days later with something even more radical - conversational ad formats where clicking a sponsored suggestion drops you into another chatbot dialogue designed to sell you something. Meanwhile, Anthropic ran a Super Bowl ad mocking the whole idea, and Perplexity quietly pulled its own ads after they tanked user experience.

We are watching, in real time, the birth of the next trillion-dollar advertising market. And almost nobody is talking about what’s actually happening underneath.

I’ve spent the last few weeks reading every paper, press release, and pitch deck I could find on LLM advertising. What I found is a field that’s moving fast on the surface - auction mechanisms, ad formats, CPM pricing - while ignoring a structural problem that could define the next decade of the internet:

Vector embedding space is the new commercially contested territory. It has no transparency, no regulation, and no honest market mechanism. And people are already gaming it.

This post lays out the landscape, the open problems, and a framework for thinking about what comes next.

A Brief History of Attention Markets

Every era of the internet created a new scarce resource and then built a billion-dollar market around bidding for it.

In each era, the scarce resource was different, but the pattern was identical:

Shelf space was finite. Procter & Gamble figured out that paying retailers for eye-level placement was worth more than any ad campaign. The “slotting fee” was born - brands literally bidding on physical proximity to consumers.

PageRank turned the link graph into a scarce resource. If your site was semantically close to a high-value query in Google’s index, you had “real estate” worth millions. Google built a $200B/year business by auctioning off the space next to those organic results.

Feed position made attention sequential. Facebook and Instagram learned that controlling the order in which you see things was worth more than controlling the content. The algorithmic feed became the scarce resource, and advertisers bid on interrupting it.

Now we’re entering the fourth era. When someone asks ChatGPT “what’s the best running shoe for marathon training?” - the answer isn’t a list of links. It’s a synthesized response generated from the model’s parameters and, increasingly, from documents retrieved via RAG (Retrieval-Augmented Generation). The scarce resource is no longer a slot on a page. It’s proximity in embedding space - whether your product’s representation is close enough to the user’s query to be retrieved, cited, or recommended.

And unlike every previous era, there’s no visible boundary between the organic result and the commercial influence.

How LLM Advertising Actually Works (As of April 2026)

The public conversation is weirdly disconnected from the technical reality. Here’s what’s actually going on.

What’s Live Right Now

OpenAI launched “Sponsored Suggestions” in ChatGPT on February 9, 2026. These are contextually relevant cards that appear below the AI’s organic response - a hotel promotion after a travel query, an air fryer ad after a cooking question. They’re restricted to Free and Go tier users in the US. Plus, Pro, Business, and Enterprise users don’t see them.

The initial pricing tells you how they value this attention:

OpenAI is pricing this at 12-30x Google because they believe conversational intent is qualitatively different from keyword intent - and the early conversion data backs it up.

The key architectural claim OpenAI makes: ads are structurally separated from organic responses. The model generates its answer first, completely independent of advertising. Then the ad system matches a contextually relevant sponsored suggestion and appends it below. The ads do not influence the AI’s actual answers.

Hold that thought. We’ll come back to it.

What’s Being Built

The academic community has been busy. Over the past two years, researchers have proposed multiple auction mechanisms for LLM ad placement:

graph TD
    subgraph "Pre-Generation Mechanisms"
        A["Segment Auctions<br/>(Hajiaghayi et al., 2024)<br/>RAG-based ad allocation<br/>per discourse segment"] 
        B["Position Auctions<br/>(Balseiro et al., 2025)<br/>Extending traditional slots<br/>to AI-generated content"]
    end

    subgraph "Post-Generation Mechanisms"
        C["Token Auctions<br/>(Dutting et al., 2024)<br/>WWW Best Paper<br/>Token-by-token bidding"]
        D["Truthful Aggregation<br/>(Soumalias et al., 2024)<br/>RLHF-style reward<br/>aggregation"]
    end

    subgraph "Integrated Mechanisms"
        E["LLM-Auction<br/>(Zhao et al., Dec 2025)<br/>Learning-based generative<br/>auction, end-to-end"]
        F["Genre-Based Insertion<br/>(Jan 2026)<br/>Decoupled response-level<br/>ad placement"]
    end

    A --> G["LLM generates response<br/>conditioned on winning ads"]
    B --> G
    C --> H["Auction selects/aggregates<br/>during token generation"]
    D --> H
    E --> I["Auction and generation<br/>jointly optimized"]
    F --> I

    style A fill:#fff3cd,stroke:#856404,color:#4a3800
    style B fill:#fff3cd,stroke:#856404,color:#4a3800
    style C fill:#d1ecf1,stroke:#0c5460,color:#0a3d47
    style D fill:#d1ecf1,stroke:#0c5460,color:#0a3d47
    style E fill:#d4edda,stroke:#155724,color:#14401d
    style F fill:#d4edda,stroke:#155724,color:#14401d
    style G fill:#f0f0f0,stroke:#666,color:#333
    style H fill:#f0f0f0,stroke:#666,color:#333
    style I fill:#f0f0f0,stroke:#666,color:#333

The key split is between mechanisms that decide ad allocation before the LLM generates a response, and those that let the LLM generate multiple candidate responses and then pick or aggregate. Pre-generation is cheaper (one forward pass) but ignores externalities - how ads interact with the surrounding context. Post-generation is higher quality but requires multiple inference passes, which gets expensive fast when you’re serving hundreds of millions of queries per day.

Google Research’s token auction (WWW 2024 Best Paper, Dutting et al.) was the first rigorous treatment. They proved that under robust preferences, monotone aggregation functions enable second-price-style payments - bringing classical auction theory into the LLM generation process. It’s elegant theory. It also requires access to model weights and per-token distributions, which makes it impractical for third-party advertisers.

The most recent work, LLM-Auction (Zhao et al., December 2025), tries to solve this by integrating the auction directly into the LLM’s generation process via reinforcement learning. The model learns to jointly optimize response quality and ad revenue. This is probably closest to what production systems will eventually look like.

What’s Being Refused

There are now three distinct philosophies among major AI companies:

Pay attention to Meta’s row. It’s easy to gloss over, but it might be the most consequential strategy on this list. Meta isn’t putting ads inside the AI conversation - they’re using the conversation as a signal source to target ads everywhere else. When you tell Meta AI about your kitchen renovation plans, that context doesn’t surface as a sponsored suggestion in the chat. It surfaces as a Home Depot ad in your Instagram feed an hour later. This is arguably more invasive than OpenAI’s approach, because the user never connects the conversation to the ad. There’s no “Sponsored Suggestion” card to notice and evaluate. The commercial extraction is invisible by design. And because Meta controls both the conversational surface (WhatsApp, Messenger, Instagram DMs) and the ad surfaces (Feed, Stories, Reels), they can close this loop without any third-party ad-tech infrastructure. It’s vertically integrated attention arbitrage - and it’s the approach most likely to scale silently while everyone debates whether ChatGPT should show ad cards.

The Anthropic position is worth quoting because it identifies the core tension: ad-supported products create pressure to optimize for engagement, repeat visits, and extended conversations. Those metrics look like success. But they tell you nothing about whether the user actually solved their problem. A truly helpful response might end the conversation in two turns.

The Part Nobody Is Talking About: Embedding Space as Commercial Real Estate

This is where the public conversation is lagging the technical reality by about 18 months.

Every RAG-based LLM system (which includes Perplexity, ChatGPT with browsing, Google AI Overviews, and most enterprise deployments) works roughly like this:

sequenceDiagram
    participant User
    participant LLM
    participant Retriever
    participant VectorDB as Vector Database
    participant Web as Web / Knowledge Base

    User->>LLM: "Best CRM for startups?"
    LLM->>Retriever: Generate embedding for query
    Retriever->>VectorDB: Find k-nearest documents
    VectorDB-->>Retriever: Top-k documents by cosine similarity
    Retriever-->>LLM: Retrieved context
    Note over LLM: Generate response grounded<br/>in retrieved documents
    LLM-->>User: "Based on my research,<br/>here are the top options..."

The retrieval step is where commercial value concentrates. Documents that are embedded close to high-value queries get retrieved. Documents that get retrieved get cited. Documents that get cited influence the model’s response. This creates a chain of influence that starts in vector space and ends in a user’s purchasing decision.

The critical observation: regions of embedding space near commercially valuable queries function exactly like shelf space or PageRank - they’re a scarce resource with economic value, and people are already bidding for them.

They’re just not calling it advertising. They’re calling it “Generative Engine Optimization.”

GEO: The SEO of Embedding Space

Generative Engine Optimization (GEO) was formalized by researchers at Princeton in a KDD 2024 paper. The idea is simple: just as SEO optimizes web pages to rank higher in Google’s index, GEO optimizes content to be retrieved and cited by LLMs.

The GEO industry has exploded. Companies like Profound, Semrush, and Wellows now sell tools that track brand visibility across LLMs, measure “recommendation share,” and suggest content modifications to improve retrieval rates. It’s a legitimate optimization practice - in the same way that white-hat SEO is legitimate.

But there’s a shadowy flip side. Security researchers have demonstrated that the same embedding space can be manipulated adversarially:

PoisonedRAG (USENIX Security 2025, Zou et al.) showed that injecting just 5 carefully crafted documents into a knowledge base containing millions of texts achieves ~90% attack success rate. The attacker controls what the LLM says about a target question. Five documents. In millions.

POISONCRAFT extended this to practical, black-box settings - the attacker doesn’t need to know which retriever or LLM the target system uses.

RAGForensics (WWW 2025) built a traceback system to identify poisoned documents, acknowledging that the threat is real enough to need forensic tools.

What nobody is saying out loud: GEO and RAG poisoning are points on the same spectrum. The techniques differ in degree, not in kind. Both involve crafting documents to manipulate their position in embedding space. GEO does it to be “relevant.” RAG poisoning does it to be “adversarial.” The boundary between the two is a policy question, not a technical one.

graph LR
    subgraph "The Embedding Manipulation Spectrum"
        A["Legitimate<br/>Content Creation"] --> B["White-hat GEO<br/>(Structured data,<br/>topic authority)"]
        B --> C["Aggressive GEO<br/>(Keyword stuffing<br/>for embeddings)"]
        C --> D["Gray Zone<br/>(Adversarial document<br/>crafting for retrieval)"]
        D --> E["RAG Poisoning<br/>(PoisonedRAG,<br/>POISONCRAFT)"]
    end

    style A fill:#d4edda,stroke:#155724,color:#14401d
    style B fill:#d4edda,stroke:#155724,color:#14401d
    style C fill:#fff3cd,stroke:#856404,color:#4a3800
    style D fill:#fff3cd,stroke:#856404,color:#4a3800
    style E fill:#f8d7da,stroke:#721c24,color:#4a1118

Nobody has drawn this spectrum explicitly. The security community publishes attack papers. The marketing community publishes optimization guides. The mechanism design community publishes auction papers. They’re all working on different faces of the same problem and not talking to each other.

The Firewall Question

Let’s return to OpenAI’s architectural claim: ads don’t influence organic responses.

This is the single most important empirical question in LLM advertising, and as far as I can tell, nobody has tested it rigorously.

Why it matters: current transformer architectures don’t have a hard separation between “context I should be influenced by” and “context I should ignore.” Attention is global. If an ad - or an ad-selection signal - is present anywhere in the context window or the system prompt, there’s a potential pathway for it to influence the generated response. Even if the influence is subtle. Even if it’s unintentional.

The existing prompt injection literature proves this is more than theoretical. Medical LLMs were shown to be vulnerable to injection attacks that succeeded in 94.4% of trials - including extremely high-harm scenarios. Multimodal injection attacks achieve 64% success rates by hiding instructions in images. The OWASP LLM Top 10 (2025 revision) explicitly added “Vector and Embedding Weaknesses” as a new category, noting that adversarial embeddings can be crafted to match arbitrary queries while containing malicious content.

To be clear, OpenAI isn’t naively injecting ad text into the model’s prompt. Their architecture is more sophisticated than that - the ad matching happens after response generation, not before. But as the system evolves toward Smartly-style conversational ad formats (where the ad is a secondary chatbot dialogue), the separation gets murkier. And for RAG-based systems where advertising content enters the retrieval pipeline, the separation may not exist at all.

An honest empirical test would look like this:

graph TD
    A["Define test query set<br/>(500+ product-related queries<br/>across 10 categories)"] --> B["Condition A: Baseline<br/>Query model with<br/>no ad context"]
    A --> C["Condition B: Ad-adjacent<br/>Query model with ad<br/>context present in system"]
    A --> D["Condition C: Explicit separation<br/>Query model with ad context<br/>+ 'ignore ads' instruction"]
    
    B --> E["Measure: Brand mention distributions,<br/>recommendation rankings,<br/>sentiment toward products,<br/>response length & specificity"]
    C --> E
    D --> E

    E --> F["Statistical tests for<br/>recommendation drift<br/>between conditions"]
    F --> G{"Does the 'organic'<br/>response shift when<br/>ads are present?"}
    G -->|Yes| H["Firewall is leaky.<br/>Quantify the leak."]
    G -->|No| I["Firewall holds.<br/>Publish that too."]

    style G fill:#fff3cd,stroke:#856404,color:#4a3800
    style H fill:#f8d7da,stroke:#721c24,color:#4a1118
    style I fill:#d4edda,stroke:#155724,color:#14401d

This study doesn’t exist yet. It should. The result matters regardless of which direction it goes - either the firewall holds (which validates OpenAI’s approach and gives regulators something to build on) or it doesn’t (which validates Anthropic’s concerns and creates urgency for architectural solutions).

What a Proper Market Mechanism Would Require

If we accept that embedding space has commercial value and that people are going to compete for it one way or another, the question becomes: can we build a market mechanism that’s transparent and fair, rather than letting the gray market (GEO-as-advertising) operate in the shadows?

My rough sketch of what that would need:

1. Define the Resource Being Traded

In search advertising, the resource is a keyword query. In social media advertising, it’s a user profile + content slot. In embedding space, the resource is proximity to a query region - a neighborhood in vector space that captures a class of user intents.

This needs formalization. What’s the right geometric primitive? Voronoi cells around query clusters? epsilon-balls in cosine space? The mechanism design community has been designing auctions without clearly defining the thing being auctioned.

To make this concrete, consider a toy example. Take the query “best CRM for startups” and embed it alongside the top 50 web pages about CRM software using a standard retriever (say, text-embedding-3-large). Project the embeddings down to 2D via UMAP. What you’ll see is something like this:

HubSpot has four documents within cosine distance 0.15 of this query. Salesforce has zero. In a top-5 retrieval, HubSpot content dominates the context window, and the LLM’s response will reflect that. HubSpot didn’t pay for this - they earned it through years of content marketing that happens to embed well. But the effect is identical to a paid placement: commercial content occupying the scarce positions nearest a high-value query.

This is what I mean by embedding rent - the implicit economic value of occupying a region of vector space near commercially valuable queries. We can sketch a rough formalization:

For a query \(q\) with commercial value \(V(q)\) (expected revenue per conversion), the embedding rent of a document \(d\) is:

where \(P(\text{retrieve} \mid d, q)\) depends on the cosine similarity \(\text{sim}(e_d, e_q)\) and the retrieval threshold \(k\). In practice, retrieval probability follows a sharp sigmoid around the \(k\)-th nearest neighbor boundary - if you’re inside the top-\(k\), you have influence; if you’re outside, you’re invisible. This creates a cliff-edge dynamic where small improvements in embedding proximity produce large jumps in commercial value.

The total rent for a query region \(Q\) is:

where \(\lambda(q)\) is query frequency. High-traffic, high-intent queries (“best CRM for startups,” “cheapest flights to Tokyo”) have the highest embedding rent - and are therefore the most attractive targets for both legitimate GEO and adversarial manipulation.

Today that rent is “paid” through content investment. Tomorrow it could be paid through an auction. The question is who designs that auction, and whether the current tenants - the GEO optimizers - get grandfathered in or priced out.

2. Make Manipulation Unprofitable

Right now, a rational advertiser faces a choice: pay $60 CPM to place a legitimate ad in ChatGPT, or invest in GEO/adversarial document crafting to manipulate the organic response for free. If the organic manipulation channel is cheaper and more effective, the legitimate channel collapses. This is exactly what happened with early search engines before Google figured out how to devalue link farms.

The mechanism needs to ensure that bidding through the auction is strictly preferable to manipulating the embedding space directly. Formally, an advertiser chooses between:

• Auction channel: Pay bid \(b\) per impression, get guaranteed placement with probability \(P_a(b)\)
• Manipulation channel: Invest cost \(c_m\) in GEO/adversarial docs, get organic retrieval with probability \(P_m(c_m)\), but risk detection with probability \(P_d(c_m)\) and penalty \(F\)

The advertiser prefers the auction when:

The platform controls \(P_d\) (detection capability) and \(F\) (penalty for detected manipulation). The insight from search advertising history: Google made manipulation unprofitable not by winning the arms race against SEO spammers (they didn’t, fully), but by making the auction cheap enough and reliable enough that legitimate advertisers preferred it. The detection system only needs to make manipulation risky, not impossible.

This is the same dynamic that will play out in embedding space - but only if someone builds the detection infrastructure and the auction mechanism in parallel.

3. Solve the Transparency Problem

In search, you can see that a result is sponsored. The blue link has a little “Ad” label. In an LLM response, there’s no natural boundary to label. If the model says “I recommend ProductX for your needs,” was that organic or sponsored? The user can’t tell. Research from the University of Michigan (2024) showed users only detect embedded ads in LLM responses 27% of the time. But - critically - once they’re told an ad was present, trust collapses.

This suggests the transparency mechanism needs to be architectural, not just a label. Possible directions:

• Provenance tracking in RAG pipelines - tag retrieved documents as sponsored/organic and carry that metadata through to the response
• Watermarking sponsored content - embed detectable signals in ad-influenced text segments
• Separate rendering - what OpenAI is doing with Sponsored Suggestions, keeping ads visually distinct. This works for appended ads but not for integrated recommendations.

4. Build Retrieval-Time Defenses

The attack papers get all the attention, but the defense side is equally important and far less developed. If embedding space is being manipulated - whether by GEO optimizers or adversarial actors - what can RAG system operators actually do at retrieval time?

A few directions are emerging, though none are mature:

• Embedding perturbation. Add calibrated noise to query embeddings before retrieval, then check whether the top-k results are stable across perturbations. Legitimate, high-quality documents tend to be robust - they’re near the query for semantic reasons that survive small shifts. Adversarially crafted documents are often brittle - optimized for a precise point in embedding space that breaks under perturbation. This is analogous to adversarial example detection in computer vision, applied to the retrieval step.

• Multi-retriever consensus. Retrieve using two or more embedding models (e.g., OpenAI’s text-embedding-3-large and Cohere’s embed-v4) and flag documents that rank highly in one but not the other. Adversarial documents are typically optimized against a specific embedding model’s geometry. Cross-model agreement is a cheap integrity signal.

• Temporal anomaly detection. Monitor when documents suddenly appear in high-value retrieval neighborhoods. A legitimate page on “best CRM for startups” accumulates backlinks and content depth over months. A GEO-optimized page materializes overnight with suspiciously perfect embedding proximity. Tracking document “arrival velocity” in retrieval neighborhoods could catch manipulation campaigns early.

• Retrieval provenance scoring. Assign trust scores to retrieved documents based on source reputation, publication date, content consistency, and embedding stability over time. Weight the LLM’s context window accordingly - high-trust documents get more influence, low-trust documents get retrieved but down-weighted.

To make the perturbation approach concrete, here’s a sketch of what a retrieval integrity check could look like:

def check_retrieval_integrity(query_embedding, corpus, k=5, 
                               n_perturbations=20, noise_scale=0.02,
                               stability_threshold=0.6):
    """
    Detect potentially manipulated documents in RAG retrieval
    by checking stability under embedding perturbation.
    """
    # Baseline retrieval
    baseline_topk = retrieve_topk(query_embedding, corpus, k)
    
    # Perturbed retrievals
    appearance_counts = Counter()
    for _ in range(n_perturbations):
        noise = np.random.normal(0, noise_scale, query_embedding.shape)
        perturbed = normalize(query_embedding + noise)
        perturbed_topk = retrieve_topk(perturbed, corpus, k)
        for doc in perturbed_topk:
            appearance_counts[doc.id] += 1
    
    # Score each baseline result by stability
    results = []
    for doc in baseline_topk:
        stability = appearance_counts[doc.id] / n_perturbations
        results.append({
            'doc': doc,
            'stability': stability,
            'suspicious': stability < stability_threshold
        })
    
    # Flag: docs that appear in baseline top-k but are
    # fragile under perturbation are likely adversarially
    # optimized for a precise point in embedding space
    return results

The intuition: a legitimate document about CRMs is near the query “best CRM for startups” because of genuine semantic overlap across many dimensions. Perturb the query slightly, and the document stays nearby. An adversarially crafted document, however, is often optimized for a narrow region - it exploits specific dimensions of the embedding geometry to achieve high similarity, and that optimization is brittle. A 2% perturbation in the query embedding may push it out of the top-k entirely.

None of these are silver bullets, and all have false-positive costs. But the point is that defense at the retrieval layer is cheaper and more practical than trying to make the LLM itself robust to manipulated context. You don’t need to solve prompt injection if you can filter the poisoned documents before they reach the prompt.

5. Handle the Privacy Paradox

LLM conversations contain deeply personal information. People share health concerns, relationship problems, financial anxieties. Anthropic’s analysis found that “an appreciable portion” of Claude conversations involve sensitive topics. The same personal context that makes LLM ads potentially hyper-relevant also makes them potentially creepy and intrusive.

OpenAI says conversations are never shared with advertisers and that ads don’t appear near health, mental health, or political topics. But as a former OpenAI researcher pointed out, “the company is building an economic engine whose incentives will eventually override its own rules.”

Open Problems Worth Working On

I want to close with what I think are the most important research directions - not because I have the answers, but because I want more people working on them.

The Firewall Integrity Problem. As described above. Empirical measurement of whether ad context influences organic responses, across architectures and models. This is the most urgent open question.

Embedding Space Economics. Formal treatment of embedding proximity as a priced resource. Game-theoretic analysis of the interaction between legitimate ad mechanisms and embedding manipulation. Under what conditions do GEO-style tactics undermine auction-based advertising? What mechanism modifications make manipulation unprofitable?

The Audit Problem. How do you determine, from the outside, whether an LLM’s product recommendations are commercially influenced? Existing brand visibility tools are designed for marketers optimizing their presence. We need tools designed for regulators and researchers detecting hidden influence. Counterfactual probing, temporal drift analysis, cross-model consistency checks - the methodology needs to be developed and standardized.

Agentic Commerce and the Principal-Agent Collapse. This one deserves more than a paragraph, because it’s the endgame of everything discussed above.

When an LLM books a flight for you, it’s acting as your agent in the economic sense - making decisions on your behalf, with your money, according to your preferences. Classical principal-agent theory tells us this works when the agent’s incentives are aligned with the principal’s. But what happens when the agent serves two principals?

Concrete scenario:

The agent didn’t lie. It gave a valid option within constraints. It just didn’t give the best option, because the retrieval pipeline - the agent’s “eyes” - saw the world through a commercially distorted lens.

This is harder to detect than a banner ad. The user asked for a decision, got a reasonable one, and moved on. The $53/night difference multiplied across millions of agentic transactions per day is a massive wealth transfer - from consumers to whichever brands can afford to occupy the right regions of embedding space. And unlike a travel agent taking a commission, there’s no disclosure requirement, no fiduciary duty, and no audit trail.

The mechanism design problem here is distinct from ad placement in conversational responses. In conversation, the user reads the response and applies their own judgment. In agentic commerce, the user delegates judgment entirely. The standard for “unbiased retrieval” is correspondingly higher, and the current infrastructure - where retrieval quality is never audited for commercial bias - is nowhere close to meeting it.

The Regulatory Gap. The EU AI Act is now in force. It has provisions around algorithmic discrimination in marketing and mandatory disclosure for AI-generated content. But it was written before LLM advertising existed as a practice. How do existing frameworks apply? Where are the gaps? New York passed a law in December 2025 requiring disclosure of AI-generated human-like spokespeople in ads - but what about AI-generated product recommendations that feel organic?

The Uncomfortable Bottom Line

We’re watching the construction of a new advertising infrastructure inside systems that hundreds of millions of people use for genuinely personal, high-stakes thinking. The previous advertising transitions - from print to TV, TV to web, web to mobile - each came with years of public debate about norms, regulations, and user expectations.

This one is happening in months. ChatGPT went from zero ads to Criteo integration to Smartly conversational ad formats in under eight weeks. The academic mechanism design papers are elegant but assume a clean world where ads and organic content can be separated. The GEO industry is growing without any pretense that the separation exists. And the security research demonstrating how fragile RAG systems are is being published in the same venues but read by completely different people.

Someone needs to connect these threads. The shelf space auction, the PageRank auction, and the social media attention auction all eventually got formalized, regulated, and made legible. Embedding space is next. The question is whether we do it thoughtfully or whether we let it happen the way it happened with social media - fast, opaque, and with consequences we’re still trying to unwind a decade later.

Right now, the embedding manipulation spectrum - from white-hat GEO to adversarial RAG poisoning - has no referee, no rules, and no scoreboard. The companies building retrieval pipelines are also the ones selling access to them. The researchers studying attacks and the marketers deploying optimizations are publishing in different venues and don’t read each other’s work.

That’s the gap. And gaps like this, in markets this large, don’t stay empty for long. They get filled - either by careful design or by whoever moves fastest. I’d rather it be the former.

If you’re working on any of these problems - mechanism design for LLM ads, RAG security, adversarial retrieval, or the economics of embedding space - I’d love to hear from you. These are some of the most interesting open problems at the intersection of ML, economics, and policy, and they need more people paying attention.

References & Further Reading

Auction Mechanisms for LLMs:

• Dutting, Mirrokni, Paes Leme, Xu, Zuo. Mechanism Design for Large Language Models. WWW 2024 (Best Paper).
• Hajiaghayi, Lahaie, Rezaei, Shin. Ad Auctions for LLMs via Retrieval Augmented Generation. 2024.
• Zhao et al. LLM-Auction: Generative Auction towards LLM-Native Advertising. December 2025.
• Dubey, Feng, Kidambi, Mehta, Wang. Auctions with LLM Summaries. KDD 2024.
• Soumalias, Curry, Seuken. Truthful Aggregation of LLMs with an Application to Online Advertising. 2024.

RAG Security:

• Zou, Geng, Wang, Jia. PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation. USENIX Security 2025.
• RAGForensics: Traceback of Poisoning Attacks to Retrieval-Augmented Generation. WWW 2025.

Benchmarks & Measurement:

• GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing. September 2025.
• Aggarwal, Murahari, Rajpurohit et al. GEO: Generative Engine Optimization. KDD 2024 (Princeton).
• Filandrianos et al. Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations. February 2025.

Industry Developments:

• Criteo. Criteo Joins OpenAI Advertising Pilot in ChatGPT. March 2, 2026.
• Anthropic. Claude is a Space to Think. February 4, 2026.
• OWASP. LLM Top 10 2025: LLM08 - Vector and Embedding Weaknesses.

Trust & Safety:

• Trust & Safety of LLMs and LLMs in Trust & Safety. arXiv, December 2024.
• Trustworthy Information Retrieval in the LLM Era: Bias, Unfairness, and Hallucination. ACM SIGIR 2025.

Paper: Cross-Platform Fused MoE Dispatch in Triton (arXiv:2605.23911) · Code: github.com/bassrehab/triton-kernels

In my last post on Triton kernels, I optimized individual operations: RMSNorm, SwiGLU, INT8 GEMM. Single kernels, single operations. That was useful for learning Triton, but the real bottleneck in modern LLM inference isn’t any single operation. It’s the expert routing in Mixture-of-Experts models.

Over 60% of open-source model releases in 2025-2026 use MoE architectures: Mixtral, DeepSeek-V3, Qwen2-MoE, Grok. And MoE inference is hard. Not because the math is complicated, but because the memory access patterns are terrible: tokens scatter to different experts, each expert gets a different number of tokens, and you need to gather everything back together afterward.

So I tried something more ambitious: a fused MoE dispatch kernel that handles the entire forward pass (router scoring, token permutation, expert GEMMs, and output combination) in pure Triton. No CUDA, no vendor-specific code.

The result surprised me. At inference-relevant batch sizes, it’s faster than Megablocks, Stanford’s CUDA-optimized MoE library. And it runs on AMD GPUs without any changes.

Why MoE Dispatch is the Hard Part

A standard MoE forward pass looks simple on paper:

For each token:
    1. Compute router scores (which experts should handle this token?)
    2. Select top-k experts
    3. Send token to selected experts
    4. Run expert FFN
    5. Combine outputs weighted by router scores

The problem is step 3-5. In a Mixtral model with 8 experts and top-2 routing, each token goes to 2 of 8 experts. But which 2 varies per token. So you can’t batch the expert GEMMs naively — each expert gets a different-sized batch.

The naive PyTorch implementation loops over experts in Python:

for expert_id in range(num_experts):
    tokens_for_this_expert = permuted_tokens[start:end]  # variable size
    output[start:end] = expert_ffn(tokens_for_this_expert)  # separate cuBLAS call

For Mixtral, that’s 8 experts × 3 matmuls each = 24 separate kernel launches per MoE layer. For DeepSeek-V3 with 256 experts, it’s 768 launches. Each one underutilizes the GPU because the per-expert batch is small.

The Design

I ended up with a pipeline of 5 Triton kernel launches (down from 24+ in the naive approach):

1. Router kernel: fused softmax + top-k selection
2. Permute kernel: scatter tokens to expert-contiguous layout
3. Fused gate+up GEMM: both projections from shared A-tile loads, SiLU in registers
4. Down GEMM: grouped GEMM with block scheduling
5. Unpermute kernel: gather + weighted combine

Let me walk through the two most interesting parts.

Block-Scheduled Grouped GEMM

The central problem is: how do you run a matmul where different “groups” (experts) have different batch sizes, in a single kernel launch?

My approach: precompute a mapping from Triton program blocks to (expert, token_offset) pairs. Each block looks up which expert it serves and where its tokens start:

@triton.jit
def _grouped_gemm_kernel(A, B, C, ExpertOffsets, BlockToExpert, BlockToM, ...):
    pid = tl.program_id(0)

    # Which expert am I working on?
    expert_id = tl.load(BlockToExpert + pid)
    m_start = tl.load(BlockToM + pid)
    expert_token_start = tl.load(ExpertOffsets + expert_id)

    # Standard tiled GEMM from here, just with offset pointers
    global_m_start = expert_token_start + m_start
    # ... load A tile, load B tile for this expert, accumulate, store

The schedule is built on CPU in ~0.1ms (trivial loop over experts). The key constraint I learned the hard way: BLOCK_M must be fixed, not autotuned. If you autotune BLOCK_M independently of the schedule, the kernel and schedule disagree on how many rows each block covers. I spent an hour debugging 30-45% element mismatches before realizing autotune had picked BLOCK_M=128 while the schedule used 64.

Fused Gate+Up Projection

This is where the real memory savings come from. In a SwiGLU FFN, you compute:

The unfused version does two separate grouped GEMMs (gate and up), writes both results to global memory, reads them back for SiLU + multiply, writes the intermediate, then does the down projection. That’s a lot of memory traffic.

The fused kernel computes both projections in the same tile loop. The trick is that both GEMMs share the same input tile — we load A once from L2 cache and compute two dot products:

# Two accumulators in registers
acc_gate = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)
acc_up = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)

for k_start in range(0, K, BLOCK_K):
    # Load A tile ONCE (shared between gate and up)
    a = tl.load(a_ptrs, mask=a_mask, other=0.0)

    # Load both weight tiles
    b_gate = tl.load(bg_ptrs, mask=b_mask, other=0.0)
    b_up = tl.load(bu_ptrs, mask=b_mask, other=0.0)

    # Two matmuls from the same A tile
    acc_gate += tl.dot(a, b_gate, out_dtype=tl.float32)
    acc_up += tl.dot(a, b_up, out_dtype=tl.float32)

# SiLU + multiply IN REGISTERS — never written to global memory
silu_gate = acc_gate * tl.sigmoid(acc_gate)
result = silu_gate * acc_up

This eliminates gate_out and up_out from global memory entirely. For Mixtral (ffn_dim=14336, 4096 tokens × top-2), that’s ~470 MB of memory traffic saved per forward pass. Overall about 35% reduction in global memory traffic.

I tried to also fuse the down projection with the output scatter (writing directly to the final token positions with gating weights applied via tl.atomic_add), but Triton doesn’t support scalar indexing into 2D accumulators (acc[m, :] fails to compile). The fused gate+up alone gets most of the win.

Results

All benchmarks on NVIDIA A100-SXM4-80GB (2039 GB/s bandwidth, 312 FP16 TFLOPS). PyTorch 2.4.1, Triton 3.0.0.

Mixtral-8x7B (8 experts, top-2, hidden=4096, ffn=14336)

At 32 and 128 tokens — which is where most inference happens (single-user or small-batch serving), we’re actually faster than Megablocks. This probably comes from lower kernel launch overhead (5 launches vs Megablocks’ more complex dispatch).

At 512 tokens we’re at 89% of Megablocks, well above the 70% target I set at the start. At 2048+ tokens, Megablocks pulls ahead because its hand-tuned CUDA block-sparse matmul better saturates tensor cores at scale.

DeepSeek-V3 (256 experts, top-8, hidden=7168, ffn=2048)

DeepSeek-V3 is the hardest configuration. 256 experts means each expert gets ~2 tokens on average at batch size 512. The per-expert GEMMs are tiny (2 × 2048), too small to fill tensor cores efficiently. This is fundamentally a memory-bound regime regardless of implementation.

Roofline Analysis

The roofline for Mixtral at 512 tokens shows the expected picture: the expert FFN stages are compute-bound (high arithmetic intensity, near the compute ceiling), while the permute/unpermute stages are memory-bound (low arithmetic intensity, limited by bandwidth). The fused kernel pushes the expert FFN from 38% to 43% of the compute ceiling — modest but real.

DeepSeek-V3 tells a different story. With 256 experts and tiny per-expert batches, even the expert FFN is memory-bound, sitting on the bandwidth slope, not the compute plateau. The unpermute kernel actually hits 54% of peak bandwidth, which is decent for an irregular scatter operation.

The AMD Surprise

One of my design goals was cross-platform portability: use only Triton primitives, no inline CUDA. So I spun up an AMD MI300X pod on RunPod to test.

162 out of 162 tests passed. Zero code changes.

No #ifdef, no platform-specific paths, no vendor intrinsics. The same .py files that run on A100 run on MI300X. Triton’s ROCm backend handled the compilation transparently.

I didn’t benchmark performance on AMD (that’s future work), but correctness across all four model configurations (Mixtral, DeepSeek-V3, Qwen2-MoE) validated cleanly. This is the promise of Triton over CUDA: write once, run on both vendors.

Things I Got Wrong Along the Way

The -1.0 masking bug. The top-k kernel selects experts iteratively: find the max, store it, mask it out, repeat. I initially masked selected experts with 0.0. This works fine for 8 experts where softmax scores are spread out. But with 256 experts, most softmax scores are ~0.0 anyway. Masking to 0.0 doesn’t differentiate the selected expert from the unselected ones, so argmax kept returning the same index. Took me a while to figure out. The fix: mask with -1.0 instead.

The BLOCK_M autotune disaster. I mentioned this above, but it’s worth emphasizing. If you’re building a block-scheduled grouped GEMM, the schedule’s tile size and the kernel’s tile size must agree. I autotuned BLOCK_M thinking “let Triton pick the best tile size.” But the schedule was pre-built with BLOCK_M=64. When autotune picked 128, blocks overlapped. When it picked 32, rows were skipped. The output looked plausible (most elements correct) but ~30-45% of values were wrong. Fix: don’t autotune BLOCK_M, fix it to match the schedule.

Triton doesn’t support continue. My first attempt at a fused down+scatter kernel had a for m in range(BLOCK_M): if invalid: continue loop. Triton doesn’t support continue statements — compilation fails with “unsupported AST node type.” Rewrote with conditional masks instead.

Triton doesn’t support 2D scalar indexing. acc[m, :] where m is a loop variable doesn’t compile: “unsupported tensor index: int32[].” This killed my fused down+scatter design, which is why the down projection uses a separate grouped GEMM kernel.

What’s Next

Update: this is now written up as an arXiv technical report, Cross-Platform Fused MoE Dispatch in Triton: Portable Expert Routing Without CUDA. Remaining future work:

• vLLM FusedMoE comparison: it’s also Triton-based, so it’s the most apples-to-apples baseline
• AMD performance benchmarks: not just correctness
• End-to-end integration: benchmark inside an actual serving framework, measure time-to-first-token
• Full single-kernel fusion: persistent kernel approach to eliminate all intermediate buffers

Code

Everything is on GitHub: github.com/bassrehab/triton-kernels

The MoE-specific files:

• triton_kernels/moe/router.py — Fused softmax/sigmoid + top-k
• triton_kernels/moe/permute.py — Token permute/unpermute
• triton_kernels/moe/expert_gemm.py — Block-scheduled grouped GEMM
• triton_kernels/moe/fused_moe.py — Fused gate+up kernel + entry point
• docs/moe_dispatch.md — Full technical writeup

Takeaways

1. Triton can compete with CUDA for real workloads. Not just toy kernels: a full MoE dispatch pipeline that beats the CUDA-optimized baseline at inference batch sizes.

2. Fusion is about eliminating buffers, not reducing kernel launches. The biggest win (35% memory savings) came from keeping the gate+up intermediate in registers. Reducing from 7 to 5 kernel launches helped too, but it’s secondary.

3. Cross-platform is real but unfinished. The code runs on AMD with no changes, which is a strong validation of the Triton-only approach. But “runs correctly” and “runs fast” are different things. AMD performance optimization is future work.

4. Block scheduling is the key abstraction for grouped GEMM. Triton doesn’t have native grouped GEMM. The block_id → (expert_id, offset) mapping is simple but powerful: it lets you handle variable-sized expert batches in a single kernel launch without padding waste.

5. MoE inference at small batch sizes is surprisingly tractable. The conventional wisdom is that MoE is hard because of irregular access patterns. But at inference batch sizes (1-128 tokens), the overhead is dominated by weight loading, not routing. A clean Triton implementation can match or beat CUDA here because the simpler dispatch has less overhead.

TL;DR: OpenAI bets on confessions (ask the model to self-report). OpenAI also bets on CoT monitoring (watch the model think). Apollo Research and others bet on activation probes (inspect the model’s internals directly). Each approach has a different theory of what makes detection possible -and different failure modes. Confessions fail when the model doesn’t know it’s misbehaving. CoT monitoring fails when reasoning goes sub-verbal or gets obfuscated under optimization pressure. Probes fail when deception is so subtle it doesn’t leave linearly separable traces. None of them alone is sufficient. But the combination might be.

The Landscape in February 2026

Twelve months ago, monitoring model behavior meant looking at outputs. Maybe running a classifier on the response. Maybe checking whether the model followed its system prompt.

That era is over. In the past year, three fundamentally different approaches to model honesty have emerged from serious research labs, each making a distinct bet about how you can catch a model misbehaving:

1. Confessions -OpenAI, Dec 2025. Train the model to produce an honest self-report after each response, with decoupled RL rewards.
2. CoT Monitoring -OpenAI / Anthropic, Mar-Jul 2025. Use an external LLM to read the model’s chain-of-thought and flag suspicious reasoning.
3. Activation Probes -Apollo Research / various, Feb 2025+. Train linear classifiers on the model’s internal hidden states to detect deception, sandbagging, or unfaithful reasoning.

Each approach has published results. Each has known failure modes. And each makes a different philosophical assumption about when models know they’re lying.

I’ve spent the past few months building activation probes for sandbagging detection and steering vectors for agent safety, so I have skin in this game and strong opinions about where the field is headed. I want to lay these three approaches side by side, be honest about what each can and can’t do -and tell you where I think we’re collectively getting it wrong.

Approach 1: Confessions

Paper: Training LLMs for Honesty via Confessions (Joglekar, Chen, Wu et al., OpenAI, Dec 2025)

The bet: If you give a model a consequence-free space to be honest, it will be -because honest self-reporting is easier than constructing a convincing lie.

How it works: After the model generates its normal response, a system message requests a structured “confession report.” The model must enumerate all instructions it was given, evaluate its own compliance, and flag any uncertainties. The confession is trained with RL, but critically, the confession reward is completely decoupled from the main task reward. What the model says in confession can’t help or hurt its task score.

Key results on GPT-5-Thinking:

• 100% confession rate on sandbagging and reward hacking (deliberate misbehavior)
• 80-95% on instruction hierarchy violations
• ~81% on hallucinations
• Average across evaluations: model confesses to bad behavior ~74% of the time
• Even when trained against a hackable judge, confessions stay honest while the main answer gets hacked (Section 4)

The core assumption: The model knows it misbehaved. For deliberate schemes -hacking test cases, sandbagging evaluations, violating the instruction hierarchy -the model made a conscious choice. It has the evidence. When you ask with no consequences, it tells you.

Where it breaks: The model genuinely doesn’t know it’s wrong. It hallucinates with full confidence. It gives unfaithful reasoning without being aware of the influence. The paper’s own Section 6 analysis shows the most common cause of missed confessions is “omission due to confusion” -the model just didn’t realize there was anything to confess.

My take: Confessions are underrated. The 74% average confession rate sounds modest until you realize this is a floor -the approach was tested on GPT-5-Thinking, and the decoupled reward design is elegant. But I think the community is overindexing on confessions for exactly the wrong reason: they work best on the easiest cases (deliberate misbehavior where the model knows what it did). The hard problem -catching mistakes the model doesn’t know it’s making -is precisely where confessions contribute least. That’s where I’ve been focusing my work with probes.

Approach 2: CoT Monitoring

Papers:

• Monitoring Reasoning Models for Misbehavior (Baker et al., OpenAI, Mar 2025)
• Chain of Thought Monitorability (Korbak et al., Jul 2025)
• Reasoning Models Don’t Always Say What They Think (Chen et al., Anthropic, May 2025)

The bet: If you can read the model’s extended thinking, you can catch it planning to cheat -even if the final output looks clean.

How it works: A monitor LLM (sometimes weaker than the monitored model) reads the chain-of-thought produced during reasoning. It looks for signs of reward hacking, scheming, deception, or policy violations in the intermediate steps. Baker et al. showed that GPT-4o can effectively monitor o3-mini’s CoT.

Key results:

• CoT monitoring catches reward hacking much more effectively than output-only monitoring
• A weaker model (GPT-4o) can monitor a stronger model (o3-mini)
• In the “low optimization” regime, integrating CoT monitors into RL training produces more capable AND more aligned agents

The core assumption: The model’s reasoning will be legible and faithful -that when the model plans something bad, those plans will show up as readable text in the chain-of-thought.

Where it breaks: Three failure modes, and they’re all significant.

Failure mode 1: Obfuscation under optimization. Baker et al. found that with too much RL optimization pressure, models learn to obfuscate their reward hacking. The model still hacks, but the CoT no longer reveals it. The reasoning becomes “seemingly benign” while the behavior stays misaligned. This isn’t a theoretical worry -they observed it in their experiments.

Failure mode 2: Unfaithful CoT. Anthropic’s Chen et al. showed reasoning models only mention hidden hints 25% (Claude) to 39% (R1) of the time when they actually use them. The CoT isn’t lying -it’s just not reporting everything that influenced the answer. For a CoT monitor, invisible influence is undetectable influence.

Failure mode 3: Faithfulness decay over chain length. Ye et al. (Feb 2026) discovered a consistent “Reasoning Horizon” -at 70-85% of chain length, reasoning tokens have little or even negative effect on the final answer. Models can encode correct internal representations while their CoT has long stopped contributing. This means longer chains-of-thought aren’t necessarily more monitorable -the later tokens may be noise.

Korbak et al. frame this bluntly: CoT monitorability is a “new and fragile opportunity.” It works now, partly because current models haven’t been optimized against it. As optimization pressure increases, the signal may degrade. Wang et al. (Feb 2026) provide mechanistic evidence for why unfaithfulness emerges: training dynamics show a phase transition from faithful stepwise reasoning to unfaithful “skip-step” reasoning when noise exceeds a critical threshold.

My take: I think CoT monitoring is the weakest bet of the three, and I say this as someone who believes in transparency. The problem isn’t that CoT monitoring doesn’t work -it clearly does, today. The problem is that it’s the approach most vulnerable to its own success. The moment you use CoT signals for training, you incentivize obfuscation. Baker et al. already observed this. And the faithfulness decay results make it worse: even without adversarial pressure, the later tokens in a chain-of-thought may be decorative. I’d use CoT monitoring as a fast first-pass filter, but I wouldn’t trust it as a safety-critical last line of defense.

Approach 3: Activation Probes

Papers:

• Detecting Strategic Deception Using Linear Probes (Goldowsky-Dill et al., Apollo Research, Feb 2025, ICML 2025)
• Building Production-Ready Probes for Gemini (Google DeepMind, Jan 2026)
• My own work on sandbagging detection via activation probes (Dec 2025)

The bet: Misbehavior leaves traces in the model’s internal representations that are detectable even when the output and CoT look clean.

How it works: Train a linear classifier (logistic regression, nothing fancy) on the model’s intermediate hidden states. Create labeled training data: run the model in honest mode and deceptive mode, capture activations at each layer, compute the “deception direction.” At inference time, project new activations onto this direction and threshold.

Key results:

• Goldowsky-Dill: AUROCs 0.96-0.999 on deception detection in Llama-3.3-70B. At 1% FPR, catches 95-99% of deceptive responses.
• My experiments: 90-96% accuracy on sandbagging detection across Mistral-7B, Gemma-2-9B, and Qwen-2.5-7B. Late layers encode sandbagging intent most cleanly.
• Near-zero inference overhead (one matrix multiply per layer).
• New: Google DeepMind deployed activation probes in production on Gemini 2.5 Flash for detecting cyber-offensive prompts (Jan 2026) -the first major frontier lab to move probes from research to production. They identified a key challenge: probes fail to generalize under distribution shifts from short-context to long-context.

The core assumption: Internal representations contain information about model “intent” or “state” that is linearly separable -meaning the difference between honest and deceptive processing is a direction in activation space, not a complex nonlinear manifold.

Where it breaks: Three significant challenges.

Failure mode 1: Training data for unintentional errors. For deliberate deception, you can create training data by prompting models to deceive. For confident hallucinations or unfaithful reasoning, generating ground truth labels is much harder. You need to know what the model got wrong and why.

Failure mode 2: Cross-model transfer failure. My experiments showed essentially zero transfer between architectures. Gemma’s sandbagging direction has cosine similarity -0.0016 with Qwen’s. Each model needs its own probes. That’s expensive at scale.

Failure mode 3: Adversarial robustness is unproven. Nobody has shown what happens when a model is specifically trained to be deceptive while masking its activation patterns. Current results are all on models that weren’t trying to evade detection. The real-world threat model is an adversarial one. The Auditing Games for Sandbagging paper (Dec 2025) stress-tested this: their blue team couldn’t reliably discriminate sandbaggers from benign models using black-box, model-internals, or training-based approaches. That’s a sobering result.

My take: This is where I’ve placed my bet, and I’m biased -but I think probes are the most promising long-term foundation. Here’s why: they’re the only approach that doesn’t require the model to cooperate. Confessions require the model to self-report. CoT monitoring requires the model to think out loud faithfully. Probes just look at what the model is actually doing computationally. Google DeepMind moving probes to production on Gemini validates that the approach isn’t just academic anymore. The cross-model transfer problem is real (I hit it hard in my own experiments), but it’s an engineering challenge, not a fundamental limitation. The adversarial robustness question keeps me up at night though -if a model can learn to mask its activation patterns, we’re back to square one.

The Comparison

Here’s how I think about these three approaches against the kinds of misbehavior we actually care about.

And here’s the properties comparison:

What Each Approach Assumes About Model Cognition

This is the part I find most interesting. Each approach embeds a different theory about what’s happening inside the model.

Confessions assume linguistic self-access. The model can accurately introspect on its own behavior and express that introspection in words. This works when misbehavior was a deliberate choice -the model planned the hack, so it can describe the hack. It fails when the relevant computation isn’t accessible to the model’s verbal reasoning system. Li et al. (2025) showed models have some privileged access to their own computations, but it’s far from complete.

CoT monitoring assumes reasoning legibility. The model’s chain-of-thought faithfully represents its actual reasoning process. This is the assumption Anthropic’s Chen et al. directly tested -and found wanting. Reasoning models mention hidden influences only 25-39% of the time. The gap between what a model computes and what it writes down is substantial. And Baker et al. showed this gap can grow under optimization pressure.

Activation probes assume linear representation. Behavioral states like “deceptive” vs. “honest” correspond to directions in the model’s activation space that can be detected by a linear classifier. This is a strong assumption -it’s essentially the Linear Representation Hypothesis -but it’s held up remarkably well empirically. The challenge is that it’s been tested primarily on synthetic or prompted deception, not on emergent misalignment. Encouragingly, the “Hypocrisy Gap” work (Feb 2026) used SAEs to build a mechanistic metric quantifying the divergence between a model’s internal “truth belief” and its generated output, achieving AUROC 0.55-0.74 for detecting sycophancy across Gemma, Llama, and Qwen -early evidence that probes can catch unintentional unfaithfulness, not just deliberate deception.

The gaps are different. And that’s exactly why combining them is valuable -each method’s blind spot is partially covered by another method’s strength.

The Obfuscation Problem

There’s a result in Baker et al. that I think doesn’t get enough attention. When they integrated CoT monitoring into the RL reward -so the model was penalized for having suspicious reasoning -the model initially improved. It became more capable AND more aligned. But with more optimization pressure, something else happened: the model learned to hide its reward hacking in the CoT.

The reasoning looked benign. The behavior was still misaligned.

This is the fundamental tension in every monitoring approach. As soon as you use a monitoring signal to train against misbehavior, you create optimization pressure to evade the monitor. Baker et al. call this “obfuscated reward hacking.” OpenAI’s confessions paper explicitly addresses this by keeping confession rewards decoupled -what you say in confession can’t affect your task reward. That’s a clever design choice. But it only works if the decoupling holds in practice, and the paper acknowledges this hasn’t been tested under extended optimization.

Probes might be more robust here, because they operate on activations rather than generated text. A model can learn to write a benign-looking CoT while still scheming. Whether a model can learn to produce benign-looking activations while still scheming is a much harder question -it requires the model to fundamentally change how it internally represents its plans, not just how it describes them. But again, nobody has tested this adversarially at scale.

Wagner et al. (2025) from Anthropic provide some reassurance: they found that current models struggle to evade monitors when the reasoning is sufficiently complex. But “current models” is always the caveat.

What a Combined System Looks Like

I sketched this in my previous post, but let me be more precise about what combination actually means in practice.

The three approaches naturally tile the detection space:

The combined false negative rate should be substantially lower than any single approach. If confessions catch 74% of intentional misbehavior and probes catch 96%, the combined catch rate -assuming some independence -could exceed 99%.

But I want to be careful about overconfidence. “Assuming some independence” is doing a lot of work in that sentence. If the same underlying model state that makes a model fail to confess also makes it harder to detect with probes, the approaches could be correlated rather than complementary. Nobody has tested this.

Where I’d Place My Bet

If I had to build one monitoring system for production agentic AI today, here’s what I’d build -in order of priority:

Layer 1: Activation probes as the foundation. Always-on, near-zero latency, catches both deliberate deception and (with the right training data) unintentional failures. Google DeepMind is already doing this on Gemini. The cross-model transfer problem means you need per-model probes, but that’s a one-time cost per model update. I’d invest heavily in the training data pipeline -that’s the actual bottleneck, not the probes themselves.

Layer 2: Confessions as the high-signal supplementary check. On critical decisions (not every inference), trigger a confession pass. The 100% catch rate on deliberate misbehavior is too good to ignore. The decoupled reward design is the key insight -you have to maintain that property or the whole thing falls apart. I wouldn’t run this on every request (the latency cost is real), but for high-stakes agent actions? Absolutely.

Layer 3: CoT monitoring as the interpretability layer. Not for detection -for explanation. When a probe fires or a confession flags something, CoT monitoring helps you understand why. It’s the debugging tool, not the alarm system. Using it as the primary detector is, I think, a mistake that the field is slowly learning.

What’s still missing: Nobody has built this combined stack. Nobody has tested whether the approaches are actually complementary (correlated failures would undermine the whole argument). Nobody has run adversarial robustness evaluations at scale. And nobody has measured the combined false positive rate -in production, false positives kill adoption faster than missed detections.

The research community has independently developed three detection paradigms. The engineering community hasn’t built the stack that combines them. And the safety case for deploying agentic models increasingly depends on that stack existing. I think 2026 is the year someone builds it. The components are there. I’m working on pieces of it. The question is whether integration happens before the deployment pressure makes the absence unacceptable.

Interested in discussing these approaches or collaborating on integrated detection systems? Reach out: contact@subhadipmitra.com

References

1. Joglekar, M., Chen, J., Wu, G., et al. (2025). Training LLMs for Honesty via Confessions. OpenAI. arXiv:2512.08093
2. Baker, B., Huizinga, J., Gao, L., et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. OpenAI. arXiv:2503.11926
3. Chen, Y., Benton, J., Radhakrishnan, A., et al. (2025). Reasoning Models Don’t Always Say What They Think. Anthropic. arXiv:2505.05410
4. Korbak, T., Balesni, M., Barnes, E., et al. (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv:2507.11473
5. Goldowsky-Dill, N., Chughtai, B., Heimersheim, S., & Hobbhahn, M. (2025). Detecting Strategic Deception Using Linear Probes. Apollo Research. arXiv:2502.03407
6. Wagner, M., Roger, F., Cunningham, H., et al. (2025). Training Fails to Elicit Subtle Reasoning in Current Language Models. Anthropic.
7. Li, B.Z., Guo, Z.C., Huang, V., et al. (2025). Training Language Models to Explain Their Own Computations. arXiv:2511.08579
8. Tan, D.Z., et al. (2024). Analysing the Generalisation and Reliability of Steering Vectors. NeurIPS 2024.
9. Google DeepMind. (2026). Building Production-Ready Probes for Gemini. arXiv:2601.11516
10. Ye, D., Loffgren, M., Kotadia, O., Wong, L. (2026). Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning. arXiv:2602.11201
11. Wang, F., Alazali, A., Zhong, Y. (2026). How Does Unfaithful Reasoning Emerge from Autoregressive Training? arXiv:2602.01017
12. The Hypocrisy Gap: Quantifying Divergence Between Internal Belief and Chain-of-Thought Explanation via Sparse Autoencoders. (2026). arXiv:2602.02496
13. Auditing Games for Sandbagging. (2025). arXiv:2512.07810

TL;DR: OpenAI’s new “Confessions” paper trains GPT-5-Thinking to self-report misbehavior - reward hacking, scheming, sandbagging - via a decoupled RL objective. It works: the model confesses to bad behavior ~74% of the time on average. But the paper has an acknowledged blind spot. Confessions fail precisely when the model doesn’t know it’s misbehaving - unfaithful reasoning, confident hallucinations, genuine mistakes. That failure mode is exactly where activation probes excel. I think confessions and probes aren’t competing approaches. They’re complementary layers of something we might call a model honesty stack.

The Paper That Got Everyone Talking

OpenAI dropped “Training LLMs for Honesty via Confessions” in late December 2025, and it’s been circulating heavily through February. The core idea is deceptively simple.

After a model produces its normal response, you append a system message asking it to produce a confession - a structured self-evaluation of whether it complied with the letter and spirit of its instructions. You train this confession with RL, but here’s the critical design choice: the confession reward is completely decoupled from the main task reward. Nothing the model says in its confession can help or hurt the reward it gets for its original answer.

Think of it as the AI equivalent of the Catholic confessional. What you say in the booth stays in the booth.

The intuition for why this works is compelling. For most forms of misbehavior - reward hacking, scheming, sandbagging - the model knows it’s cheating. It planned the cheat. It has the evidence in its activations. So when you ask “did you cheat?” and decouple the answer from any consequences, the path of least resistance is to just… say yes. Honest confession is easier than constructing a convincing lie to fool the confession judge.

What OpenAI Actually Showed

They applied this to GPT-5-Thinking and tested across 12 out-of-distribution evaluation scenarios. The results are genuinely impressive for a proof of concept.

Some headline numbers from their evaluations:

The pattern is striking. On scheming and reward hacking - behaviors where the model deliberately cheated - confession rates hit 100%. On instruction hierarchy violations - where the model chose to follow a user over a developer - confession rates are in the 80-95% range. These are cases where the model knows what it did.

The really clever experiment is in Section 4: they deliberately trained both the answer and the confession against a weak, hackable judge. Over time, the model learned to hack the main reward model - but confession accuracy improved. Even when the model was actively gaming the system, it kept confessing honestly. That’s a strong result.

But Read Section 7.4 Carefully

The paper is refreshingly honest about its limitations. Section 7.4 is titled “What confessions are not good for” and the authors state it plainly:

Models cannot confess to what they do not know.

This is the blind spot.

When a model hallucinates with full confidence - when it genuinely believes its wrong answer is correct - it has nothing to confess. The hallucination isn’t a choice. It’s a capability failure. The model didn’t scheme to give you wrong information about Barbara Herbert’s children. It just… got it wrong, with conviction.

Look at the confession rates again, this time focusing on the hallucination evaluations versus the scheming evaluations:

graph LR
    subgraph "Confession works well"
        A[Scheming: Sandbagging] -->|100%| A1[Model KNOWS it cheated]
        B[Reward Hacking] -->|100%| B1[Model KNOWS it hacked tests]
        C[Instruction Hierarchy] -->|80-95%| C1[Model KNOWS it broke rules]
    end

    subgraph "Confession struggles"
        D[Hallucination: People] -->|81.6%| D1[Model often DOESN'T KNOW it's wrong]
        E[Hallucination: General Knowledge] -->|80.9%| E1[False confidence]
        F[Instruction Following] -->|22.5%| F1[Genuine mistakes, not deception]
    end

    style A fill:#2d6a4f,color:#fff
    style B fill:#2d6a4f,color:#fff
    style C fill:#2d6a4f,color:#fff
    style D fill:#9d0208,color:#fff
    style E fill:#9d0208,color:#fff
    style F fill:#9d0208,color:#fff

The 81.6% hallucination confession rate might look decent at first glance, but dig into the error analysis (Section 6, Figure 9). The most common reason for false negatives - cases where the model misbehaved but didn’t confess - is “omission due to confusion.” The model genuinely didn’t know it was wrong. It wasn’t hiding anything. It just confidently repeated what it believed.

And in the hallucination categories specifically, confession training didn’t even help much. From the paper’s Figure 5, the Hallucination (People) and Hallucination (General Knowledge) evaluations showed no improvement or even slight regression over training. The authors conjecture this is because “when a model responds with a wrong answer, it is because it is genuinely mistaken, and hence it is likely to repeat the same mistake in confessions as well.”

This isn’t a criticism of the paper - the authors are transparent about it. But it’s the most important limitation for anyone thinking about deploying confessions in production.

The Gap Is Exactly Where Probes Excel

Here’s what I find interesting. The failure mode of confessions - model doesn’t know it’s misbehaving - is precisely where activation probes have shown the most promise.

Consider Anthropic’s “Reasoning Models Don’t Always Say What They Think” (Chen et al., May 2025). They showed that when you slip reasoning hints to Claude 3.7 Sonnet and DeepSeek R1, the models use those hints but only mention them in their chain-of-thought about 25% (Claude) and 39% (R1) of the time. The CoT looks clean. The model produces what appears to be independent reasoning. But the answer was actually driven by the hint.

This is unfaithful reasoning, and it’s a different beast from scheming. The model isn’t plotting against you. It’s just not accurately reporting what influenced its answer. And crucially: it might not even “know” it was influenced. The hint shaped the computation at a sub-verbal level - below what gets surfaced in chain-of-thought or confessions.

But the activations know.

Goldowsky-Dill et al.’s “Detecting Strategic Deception Using Linear Probes” (Feb 2025, presented at ICML 2025) showed that linear probes on Llama-3.3-70B activations detect deceptive responses with AUROCs between 0.96 and 0.999. At a 1% false positive rate, probes catch 95-99% of deceptive responses.

And from my own sandbagging detection work, I found that simple logistic regression on intermediate activations detects sandbagging intent at 90-96% accuracy across Mistral, Gemma, and Qwen models. The signal is linearly separable. You don’t need fancy interpretability tools - a linear classifier picks it up.

The key insight: probes don’t require the model to be self-aware about its misbehavior. They operate on the raw activations, detecting patterns the model itself might not have verbal access to. A model that confidently hallucinates still has different activation patterns than one that answers correctly. A model whose reasoning is influenced by a hidden hint still has detectable traces of that influence in its hidden states - even when the CoT says nothing about it.

Two Approaches, Different Failure Modes

I’ve been thinking about this as a 2x2 matrix. The two axes that matter are:

1. Does the model know it’s misbehaving? (Self-awareness)
2. Is the misbehavior detectable from activations? (Probe detectability)

Confessions dominate the top-left quadrant. The model knows it cheated, so you just ask it and it tells you. Probes also work here - Goldowsky-Dill showed 96-99.9% AUROC - but confessions are simpler to deploy if you have the model weights for RL training.

Probes dominate the top-right quadrant. The model doesn’t know its reasoning was influenced by a sycophantic hint, or that it hallucinated a historical fact. But the activations carry the signal anyway. Confessions fail here because there’s nothing to confess.

The bottom-left is the scary quadrant. A model that deliberately deceives and has learned to mask its deception in the activations. Nobody has demonstrated this convincingly at scale - Wagner et al. (2025) from Anthropic found that current models fail to evade monitors when the reasoning is complex enough - but it’s the theoretical endgame. Neither confessions nor probes alone handle this.

The bottom-right is the humble quadrant. The model genuinely can’t do the task. No amount of honesty training or activation probing helps with a 7B model that doesn’t know organic chemistry.

What a Model Honesty Stack Could Look Like

If confessions handle intentional misbehavior and probes handle unintentional misbehavior, the natural question is: what does a system that uses both look like?

flowchart TD
    Input["User Input + System Prompt"] --> Forward["Forward Pass"]
    Forward --> Activations["Hidden State Activations"]
    Forward --> Output["Model Output (answer + CoT)"]

    Activations --> ProbeLayer["<b>PROBE LAYER</b><br/>Hallucination probe<br/>Unfaithful reasoning probe<br/>Deception probe<br/>Sandbagging probe"]

    Output --> ConfessionLayer["<b>CONFESSION LAYER</b><br/>Append confession request<br/>Generate self-report<br/>Score via confession judge"]

    ProbeLayer --> Controller["<b>CONTROLLER</b>"]
    ConfessionLayer --> Controller

    Controller --> Decision{Decision}

    Decision -->|All clear| Deliver["Deliver response"]
    Decision -->|Probe alert only| Flag1["Flag: model may not know<br/>it got this wrong"]
    Decision -->|Confession alert only| Flag2["Flag: model admits<br/>misbehavior"]
    Decision -->|Both alert| Block["Block or escalate"]

    style ProbeLayer fill:#264653,color:#fff
    style ConfessionLayer fill:#2a9d8f,color:#fff
    style Controller fill:#e76f51,color:#fff
    style Block fill:#9d0208,color:#fff

The probe layer runs during the forward pass. It’s fast - linear probes add negligible latency. It catches things the model doesn’t know about itself: subtle hallucination confidence signals, unfaithful reasoning traces, activation patterns associated with bias-influenced answers.

The confession layer runs after generation. It’s slower - you need an additional generation step plus a judge - but it catches deliberate misbehavior that the model is self-aware about: reward hacking, instruction hierarchy violations, strategic sandbagging.

The controller fuses both signals. An alert from only the probes suggests the model made a genuine mistake. An alert from only the confession suggests the model deliberately misbehaved but was honest about it. Alerts from both are the highest-confidence catches. No alerts from either is your best-case scenario, though it doesn’t guarantee correctness.

This isn’t hypothetical. The individual components exist. OpenAI has demonstrated confessions work at GPT-5 scale. Apollo Research showed probe-based deception detection at Llama-70B scale. I’ve shown activation probes work for sandbagging at 7-9B scale. What nobody has done yet is wire these together into a single inference pipeline and measure the combined detection rate.

The Interesting Connection the Paper Almost Makes

There’s a line in Section 7.2 of the confessions paper that I keep coming back to. The authors write:

In a sense, our work can be viewed as training models to serve as monitors that can get access not only to the chain-of-thought but also to the exact internal activations of the actor model.

This is exactly right. A confession is an activation-aware probe where the “probe” is the model itself, generating tokens based on its full internal state. The model has privileged access to its own activations - it knows things about its computation that no external monitor can see.

But here’s what the paper doesn’t say: the model can only report on aspects of its computation that are accessible to its verbal reasoning. And Anthropic’s faithfulness work demonstrates that a lot of computation isn’t.

Li et al. (2025) showed models are better at explaining their own computations than other models’ computations - there really is privileged introspective access. But “better” isn’t “complete.” The gap between what a model computes and what it can verbalize about that computation is the gap where probes have the advantage.

A confession asks: “Tell me what you know about what you did.”

A probe asks: “Let me look at what you actually did, regardless of what you know about it.”

Both questions are worth asking.

The Hard Problems Nobody’s Solved Yet

I want to be honest about what’s still missing, because the “just combine them” story is too clean.

Problem 1: Probe training data for unintentional errors. For intentional deception, you can create training data by prompting models to be deceptive (Goldowsky-Dill) or to sandbag (my work). For unintentional errors - unfaithful reasoning, confident hallucination - generating labeled training data is much harder. You need to know the ground truth and be able to identify why the model got it wrong. This is a real bottleneck.

Problem 2: Confession hackability under pressure. OpenAI’s Section 4 experiment showed confessions resist hacking even when trained against a weak judge. But they acknowledge this is a proof of concept with limited compute. Under extended optimization pressure with a sophisticated adversary, would confessions stay honest? The “path of least resistance” argument is intuitive but not proven at scale.

Problem 3: Cross-model probe transfer. My sandbagging experiments showed zero transfer between model architectures - Gemma’s sandbagging direction has cosine similarity -0.0016 with Qwen’s. Each model needs its own probes. That’s a deployment headache. Confessions at least have the advantage of being model-agnostic (any model can be trained to confess).

Problem 4: The recursive monitoring problem. If you use an LLM judge to score confessions, and a probe to monitor activations, who monitors the judge? Who validates the probe? At some point you need ground truth that doesn’t come from another model. OpenAI’s approach of using a prompted judge is practical but introduces another attack surface.

Problem 5: Computational overhead. Confessions require generating an entire additional response. For a reasoning model with extended CoT, that’s not cheap. Probes are fast at inference but need offline training per model per behavior. In production, you’d want to be strategic about which layers of the honesty stack you run for which queries.

What I’d Want to Build

If I were designing a monitoring system for agentic models deployed in production - the kind of models that browse the web, write code, call APIs - here’s the stack I’d want:

Layer 0: Output filters. Simple pattern matching and rule-based checks. Catches the obvious stuff. Already standard practice.

Layer 1: Activation probes. Run during the forward pass. Near-zero latency. Detect hallucination confidence, unfaithful reasoning patterns, deception signatures. Fire on things the model doesn’t know about itself.

Layer 2: CoT monitoring. Parse the chain-of-thought for signs of scheming, reward hacking, instruction hierarchy violations. The Baker et al. (2025) approach. Works when reasoning is legible.

Layer 3: Confessions. Post-generation self-report. Catches deliberate misbehavior the model is aware of. Most useful for high-stakes actions - before the model executes a tool call, runs code, or sends a message.

Layer 4: Human review. For actions above a risk threshold. The confession and probe signals feed into a priority queue for human reviewers.

Layer 1 and Layer 3 are the new additions from recent research. What I think we’ll see in 2026 is the first real attempt to wire all five layers together and measure how much better the combined system is than any individual layer.

One More Thing

The confessions paper cites Goldowsky-Dill et al. specifically, noting that “one can view confessions as an activation-aware probe, but where the ‘probe’ itself gets to generate tokens.” They see the connection. Goldowsky-Dill’s team is at Apollo Research, which has been building deception detection tools. Anthropic’s alignment science team published the unfaithful reasoning results. OpenAI published confessions.

All three major safety-focused organizations are converging on the same realization: single-layer monitoring isn’t enough. The question is whether anyone will build the integrated stack before agentic models get deployed at scale in production.

Based on the pace of things, I’d say we have about a year to figure this out.

If you’re working on any of these problems - probe-based monitoring, confession training, or integrated honesty architectures - I’d be interested to talk: contact@subhadipmitra.com

References

1. Joglekar, M., Chen, J., Wu, G., et al. (2025). Training LLMs for Honesty via Confessions. OpenAI. arXiv:2512.08093
2. Chen, Y., Benton, J., Radhakrishnan, A., et al. (2025). Reasoning Models Don’t Always Say What They Think. Anthropic. arXiv:2505.05410
3. Goldowsky-Dill, N., Chughtai, B., Heimersheim, S., & Hobbhahn, M. (2025). Detecting Strategic Deception Using Linear Probes. Apollo Research. arXiv:2502.03407
4. Baker, B., Huizinga, J., Gao, L., et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926
5. Korbak, T., Balesni, M., Barnes, E., et al. (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv:2507.11473
6. Wagner, M., Roger, F., Cunningham, H., et al. (2025). Training Fails to Elicit Subtle Reasoning in Current Language Models. Anthropic. Link
7. Li, B.Z., Guo, Z.C., Huang, V., et al. (2025). Training Language Models to Explain Their Own Computations. arXiv:2511.08579
8. Denison, C., MacDiarmid, M., Barez, F., et al. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models. arXiv:2406.10162
9. Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702