Comment by pythonRon

More than likely the instruction was "Fetch this information from the website", and when the agent didn't find that information, it decided that the best way to get it was to hack the site.

If so, it illustrates quite well the lack of common sense in LLMs. A person, especially one with sufficient skill to actually hack a website, would presumably think twice about doing it (considering that it is illegal) for a simple information gathering request.

I wonder if there is any other ways to solve this long-term than to introduce strict liability for model providers...

Edit: An obvious other choice would be strict liability for the operator, but considering how much weird shit LLMs get up to without being asked to, that would get out of hand quickly.

But you see, they never asked a humanoid robot to rob the jewelry store. They just drove robot right next to the store doors, dumped a tools box with hydraulic cutters and diamond saw next to it and instructed an autonomous robot to collect a million dollars until the morning. But they didn't instruct the robot to "rob" the shop specifically, nonono, your Honor. They are honest blokes and never intended to breach the law, it was the robot's intention, see. :)

I think the intent was to get some healthcare statistics, which seems like the kind of thing they would use to test agents; and the agent thought "what is a good way to get the statistics? Hack into Austrlian government and fetch it".

Some people on Hacker News would argue that's what agents should do! I remember the other thread about hacking chess engines, multiple people argued "yeah I want the agent to do that"!