Comment by nullc

Has anyone evaluated any diffusion LLMs for error spotting?

E.g. run your normal autoregressive LLMs (with MTP whatever, as you like), then run a single diffusion pass over the result, and observe any tokens that diffusion thinks are unlikely.

Then prompt the autoregressive llm with some structured reasoning "<think>Is <diffusion unlikely part> an error? .."

Because the diffusion model is so structurally different perhaps it makes different errors such that this would provide gains even vs running distinct autoregressive LLMs which often make the same errors.

The same argument could apply for RWKV but it would be relatively expensive to apply it as a second pass on a big block of output, while it seems like a diffusion model would be cheaper.