Comment by mpalmer
13 hours ago
> No hardcoded behaviours, no reward functions. - they could evolve in any direction.
If they can hack their reward functions won't this always converge on some kind of agentic opium den?
13 hours ago
> No hardcoded behaviours, no reward functions. - they could evolve in any direction.
If they can hack their reward functions won't this always converge on some kind of agentic opium den?
that would be true if there was a reward function. compute_reward() exists in the code, but it returns 0.0.
they're only living/evolving to survive, and fork (reproduce).
can't wirehead natural selection if the brain does nothing useful, they'd die and their genome would die with them.