Comment by cubefox

There is a new way to train diffusion models to insert tokens between existing tokens rather than unmasking <mask> tokens: https://openreview.net/forum?id=VbvXjs5f72

However, I believe this would "only" be able to insert tokens, not to delete tokens again it mistakenly produced before. (The deletion in the title refers to the reverse process during training, where tokens are progressively deleted rather than masked.)