Comment by dnautics

There is no reason to believe that the transformers couldn't be doing something close to what copycat does (especially with thinking tokens), as an emergent phenomenon of the sheer size of the corpus. The architecture is certainly capable of encoding the actions in copycat anyways.