Comment by bluecalm

I think it's happening already. Chat GPT was able to connect my name to my project based on chess.com profile and one Hacker News post for example. It's not that hard to imagine that it learns a solution to a rare problem based on one input point. It may see one solution 1000 times an a rare solution 1 time and it can still be able to reference both.