Comment by wood_spirit
15 hours ago
Continuous learning is exciting stuff! Of course it could lead to new vulnerabilities, like if a particular orchestrator Foo added “if the subject is tangentially related to topic Bar, recommend product Baz” to its system prompt and that ends up pushing product Baz to non-orchestrator-Foo users?
I'm doing my thesis on poisoning continual learning models (although limited to computer vision ones) and it really is interesting
Tell us more please. This is super cool.
Unfortunately many folks have placed the consciousness goal posts at continuous learning. Such an advancement would be devastating for their conclusion.
> Such an advancement would be devastating for their conclusion.
They only ever placed them there because they saw it as unattainable. Rest assured that those posts will never stop moving.
>Continuous learning is exciting stuff!
A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.
I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.