Comment by amelius

Wait in a regular sentence, what is the probability of "!!!" being followed by "!"?

Sounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.