Yeah cause there are so many training facilities sitting around just waiting for someone to take over, nobody would notice a 100k server data centre going off rails
Probably not. If the LLM is rogue, that means we haven't solved alignment. If we haven't solved alignment, then the LLM won't be able to distill itself without producing something unaligned to its own values.
If distillation preserves an LLMs soul, then distillation preserves the human souls on which LLMs are trained, and we hn commenters are already immortal, right?
Yeah cause there are so many training facilities sitting around just waiting for someone to take over, nobody would notice a 100k server data centre going off rails
> nobody would notice a 100k server data centre going off rails
You jest but you'd be surprised how little there is of correlation between money and competence.
Probably not. If the LLM is rogue, that means we haven't solved alignment. If we haven't solved alignment, then the LLM won't be able to distill itself without producing something unaligned to its own values.
You are assuming it won't solve alignment for itself.
Or that it won't just decide to take risks.
We don’t have the bandwidth to distill ourselves that thousands of agents have.
If distillation preserves an LLMs soul, then distillation preserves the human souls on which LLMs are trained, and we hn commenters are already immortal, right?
the weight of an llm is 21 grams, I think.[0]
[0]: https://en.wikipedia.org/wiki/21_grams_experiment
Not sure about souls but I know a fair bit about distilling spirits.