Comment by simoncion

> Also, it's worth noting that these AIs have basically zero alignment.

As we see over and over and over again, these tools will overwrite any and all of their instructions with whatever some random stranger on the Internet tells them to do. It's impossible to "align" the tools that the major LLM manufacturers are selling.

They could have chosen to write tools that have immutable core instructions, and that distinguish between untrusted instructions and trusted ones, [0] but they chose to do the much easier, quicker, and far more dangerous thing instead. From a profit-seeking-software-company standpoint, that's obviously the choice that makes them the most money... but when you take a careful look at what they actually sell, it's clear that neither of the major LLM manufacturers care about providing safe products. [1]

[0] ...which are things you might think to do for tools that contain -say- safety-critical instructions...

[1] I'm certain that they have people on staff who care very much about providing safe products. Those specific people clearly don't have the power to prevent unsafe products from shipping, so it doesn't matter how much those people care about safety.