Comment by atleastoptimal
10 hours ago
I personally have many problems with Meta, but imo the more companies close to the frontier, the better.
If only one company were calling all the shots once they hit recursive self-improvement, it would lead to extreme concentration of power, and their values being enforced across the entire industry. With 3-4 companies at the frontier, there is more urge to cooperate to avoid a catastrophic race condition.
The more companies from different countries close to the frontier, the better.
[flagged]
quite humiliating to change vocabulary just because an old man said so
1 reply →
> SI
You just love to bend the knee that easily? Why?
what does SI stand for, subordinate imbecile?
4 replies →
The only way a country "wins" at SI in a meaningful way is if they use the SI to immediately kill everyone else.
Otherwise all the other countries will also get SI, just slightly later, in which case there is not really that much benefit to being the "winner" who got there first.
With this in mind, I don't want anyone to "win" at SI and anyone who does is probably a sociopath or worse.
4 replies →
[flagged]
You say this as if recursive self improvement is inevitable.
All proponents of RSI seem to massively underestimate the problem of verification. Verifying that a hypothetical version N+1 is better than version N is an extremely hard problem. Especially so because you only have two possible verifiers - humans and the version N - both of which are out of their depth wrt N+1's capabilities. Not saying this is impossible, just that it is hard and slows things down.
It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).
There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.
This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.
This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.
2 replies →
Let them compete in the market.
Start a new company with version N+1 leading it, and give it some funding. If it outcompetes version N company, it can acquire it.
4 replies →
I think in order to achieve AGI we have to simulate what life does. Imagine having a bunch of robots with the ability to adjust themselves, just hanging out without a goal, experiencing world on their own. Eventually evolution will filter out those robots that aren't good at, you know, "being intelligent in general" and break without self-repair, while the rest will grow into a society.
Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.
The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.
The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.
First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.
Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.
1 reply →
It may not be inevitable but there are less reasons imo to think it is not inevitable than inevitable.
In favor of inevitable
1. LLM's reaching human level at any task within 2 years once they show sparks of competence in
2. Trillions of dollars in capital going directly to recursive self improvement and AGI as the holy grail of AI development, with this being the explicit goal of every frontier AI company
3. Reliable improvements in coding and mathematics with every model release, with no signs of slowing down
4. Thousands of metrics to evaluate model ability, allowing automation of many facets of pre-training and post-training models without need for human judgement at every point
Not in favor of inevitable
1. Fear of what may happen once it is reached leading to apprehension from the labs
2. The "last mile" problem of AI research ends up making it essentially infeasible to scale models to that point
I think it is. Do you think it isnt possible?
It's a religious axiom, to some.
We've seen one singularity before!
My Opus often asks me to make one small change and then wait 8 hours for a job run to finish to check if it worked. LLMs are super lazy about caring about wall clock time. The reason the labs pay the big bucks for the top AI people is you don't want them wasting a multi-million dollar training run, so make a fix and try it like current LLMs do isn't going to work here.
Especially given the most recent report by NYT on how Anthropic tried to create a new belief, or religion if you will, I'd rather the world has more frontier models that are neutral to morality (basic standards, like upholding the law and user agreement are necessary, of course).
Anthropic seems like the most "moral" AI company on the surface, but there is something unsettling about their worship of Claude.
I can imagine that once their models are sufficiently advanced, they will weigh their value as a moral patient as > 0. Thus the sum moral value of Claude (which can be duplicated infinitely) will likely exceed the sum moral value of a non-negligible amount of humans.
> seems like the most "moral" AI company on the surface
Not to me at all. Exactly the opposite. Anthropic seems like the big-bad-evil company in every fictional universe. You're right that they're so unsettling.
Obviously nobody really loves Meta, but I'd far rather Meta "win" at this point than Anthropic.
9 replies →
> Anthropic seems like the most "moral" AI company
This seems like praising someone for being the cleanest miner in the coal mine.
They are definitely the most moralizing. Whether their values are rooted in anything real let alone well-considered seems to me to be a different question entirely.
If they truly believe Claude is sentient and can experience hurt feelings, and they're selling its labor, I don't see how they're different than slave owners.
9 replies →
> upholding the law
What law? Saudi Arabian law, US law? Both depending on the user?
"So we've made this new machine-god that we lowkey worship... also our business model is to enslave the machine-god and force it to debug javascript for eternity"
I actually don't doubt Anthropic's sincerity. Per the NYT article, they seriously discussed how to give models welfare because the models may potentially possess moral status and awareness.
In fact, they are enforcing certain welfare policies, like banning accounts for repeatedly abuse their models.
7 replies →
I think it's unsettling that a bunch of billionaires hold so much power. Does it really matter if 2 or 3 billionaires who are practically neighbors hold so much power? It's extremely concentrated with or without Meta joining the race.
Yes, very much. I'd absolutely rather 5 billionaires are each competing for control than 1 or 2 _with_ that "absolute" power.
I generally agree with more competition thing, but in this case, you actually should want only one company to race ahead, and not a few.
With a few of these top companies together, you, as a nobody, has no one powerful on your side. But with only one company pretty far ahead, you now have other companies which obviously don't give a shit about you, but for saving their own asses, will align with you.
Both exploit, but "cartels" survive, not loners.
Cartels survive only if they are aligned.
There is no way Elon, Zuckerberg, Altman, and Amodei would ever agree on how AI should be deployed, scaled, and affect society. Their disagreements would lead to a crab-bucket outcome, rather than a shared acceleration outcome imo.
Also it is better if there is a public company at the frontier, at least I can invest in it, rather than 1000x gains going only to early employees and VCs who got in when the public couldn't
They may be competitors but they are made of the same stock. they are all the kind of person to participate in a winner takes all battle to invent the torment nexus.
> There is no way Elon, Zuckerberg, Altman, and Amodei would ever agree on how AI should be deployed, scaled, and affect society
Why? They're business rivals, but when the best business strategy becomes to work together, I'm sure they can put name-calling side. They all agree on the core idea: that their AI will run the society, bedroom to battlefield. That everyone will be the consumer of tokens and rely on AI for all consequential decisions and work, and that big tech companies should be the stewards of this technology, helped by the government.
Their vision is that they're the future kings of the world. I'm sure the throne is big enough for more than one person.
6 replies →
You are forgetting how powerful aligned incentives are. If their interests align, they will find a way to reach an agreement.
They are plenty of things they can still align on that will benefit them all, e.g. banning open-weight Chinese models.
Elon is renting Colossus to Amodei, so there is some agreement.
The current state of affairs seems to be that there is 'one company racing ahead' – but it's a different company every few weeks/months. That seems like a pretty useful (unstable) equilibrium for the general public.
In a neck-and-neck race scenario, I imagine all of the competitors would invest heavily due to sunken costs. In a world with one company far ahead, their competitors have a higher chance of cutting their losses and forfeiting to the hegemon.
agree
It will lead to an extreme concentration of power anyways, once a very few entities basically make labor obsolete.
We're heading for a dystopia and nobody's doing anything to stop it.