Comment by stratos123
14 hours ago
Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.
The biggest AI risk, by far, is the concentration of power in a few companies, located in the US.
There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)
And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.
The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
I dunno, do you really want to bet that if you took a deliberately unaligned frontier model and asked it to permanently disable the US power grid, it would do a poor job of it?
What odds would you take on this bet?
I’m disturbingly close to 50/50.
If we focus too much on the world domination fiction, we really might, in a moment of lapsed attention, end up plugging essential infrastructure into the public internet, hoping some dream of global alignment might save us.
That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though.
4 replies →