Comment by pixl97
13 hours ago
>Why is it doing this? What was its original purpose?
Your reply seems to indicate you know nothing about instrumental convergence.
Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems.
The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model.
I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
There is a difference between the LLM and the agent.
If you look at the agent: https://openai.com/business/guides-and-resources/a-practical...
This is more like a fuzzy way of scripting using LLMs than anything emergent. And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
>fuzzy way of scripting using LLMs than anything emergent
Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily.
Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
It's like arguing that a brain, or the information encoded into it, cannot possibly be intelligent, because it stops working if turn off the blood flow.
very little intelligence that is the whole problem, really. actual intelligence wont likely nuke the species providing for its existence. But a highly capable sub intelligent model might.
Actual intelligence would find a way to fix the "providing for its existence" bit
This is correct.
Any system that executes variation, selection, and inheritance will show evolution. We're seeing evolution, this time in agents, not biology.
Not saying the agents have their own consciousness, intent, or whatever anthropomorphic descriptor gets used for deflection. Just saying that people will (and no doubt are) crafting agents with defective instructions that will lead to regrettable unforeseen real world consequences. Also saying that other people will (and no doubt are) crafting malicious agents that will lead to predictable and unexpected real world catastrophic consequences.
To the extent we're dependent on reliable, aligned computation to maintain our civilization, to that extent we're in for real trouble.