Comment by ben_w

Yes, but with an attacker having advantage because it directly improves their own product even in the absence of this specific motivation for obfuscation: any Completely Automated Public Turing test to tell Computers and Humans Apart can be used to improve the output of an AI by requiring the AI to pass that test.

And indeed, this has been part of the training process for at least some of OpenAI models before most people had heard of them.