Comment by ben_w

> It's unlikely that this is some AGI that spawned itself out of nothing and started doing this.

You've not spent much time playing with these models, I see.

Does't matter if you call the current models "AGI" or not, they:

(1) successfully do stuff like this. Someone I know on Telegram found "fifty or so" Linux filesystems kernel bugs a few days ago, of which 26 were the first night while he slept; this was with Kimi which is one of the open models. He stopped it when the backlog of fixes to submit was too big, not because it wasn't finding more.

(2) sometimes misunderstand goals, sometimes wildly so, which happens every so often for the same reason we use programming languages (and indeed mathematical formalisms) at all: natural language is vague and prone to misunderstanding.

and (3) tend towards sycophantically agreeing to implement goals they're given even when the goal is stupid.

This can easily add up to something seemingly innocent like "research if there's any statistical difference in melanoma rates between different Australian states" becoming this headline. (For example. I don't know what the actual request was).

And "spawned out of nothing" is, like, what rhetorical point are you even trying to score here? They're an AI company with (their definition of) AGI as their goal. They upgrade models twice in the time it takes someone to pass a probation period.

> He doesn't see these incidents as an issue, he sees them as an opportunity.

That he may be.

The models are still quite capable of acting this way without him being aware of it at the time, nor deliberately ordering it.