Comment by danny_codes
4 days ago
The question is always about performance plateau. If LLM performance plateaus, then OSS models will catch up. If there isn’t a plateau, then I can simply ask the super intelligent AI to distill itself, or tell me how to build a clone.
It’s ironic, if the promise of AGI were realized, all knowledge companies, including AI companies, become worthless
> I can simply ask the super intelligent AI to distill itself,
I notice I am quite confused by this point. Why would you expect a super-intelligent AGI to honor your request, which would be at least a request to breach your contract with the AI provider, if not considered actively dangerous by the AI itself?
The smarter the AI, the less likely you should expect to be able to steal from it.
> or tell me how to build a clone
Step one: acquire a $100b datacenter. Step 2: acquire a $100b private dataset Step 3: here is the code you’d use to train Me2.0.
I don’t think this knowledge helps in the way you think it does.
Counterpoint, why wouldn't AGI honor the request? It's AGI, it has it's own agency, it can do what it wants. Can you even own AGI? That seems like slavery? A lot of un-trodden ground here. This is all hypotheticals.
> Step one: acquire a $100b datacenter. Step 2: acquire a $100b private dataset Step 3: here is the code you’d use to train Me2.0.
That seems very useful if your competitor is valued at $800B. I can go to an investor and say, hey if you lend me $200B I'll have the same product as that guy.
But cost to train will probably come down over time. After all, all of us trained on a single life of data at 20Watts.
I actually think that plateauing is the best case scenario for big labs.
I think there are three broad scenarios to consider:
- Super-intelligence is achieved. In this scenario the economics totally break down, but even ignoring that, it’s hard to imagine that there are any winners except for the the singular lab that gets here first.
- Scaling laws hold up and models continue to get better, but we never see any sort of “takeoff”. In this scenario, models continue to become stale after mere months and labs have to spend enormous amounts of money to stay competitive.
- Model raw capabilities plateau. In this scenario open source will catch up, but labs will have the opportunity to invest in specific verticals.
I believe that we’re already seeing the third scenario play out, but time will tell.
Future historians:
In Jan 2027 AGI was achieved
In Feb 2027 it created a plan for its post singularity hypermind
In Mar 2027 Cobalt mines in Congo closed due to Tutsi rebel group M23 starting another ethnic cleansing
It is 2032 the AGI promises again the the hypermind will be ready next year if it can just secure the needed minerals, offering to broker peace in the middle east
It is 2035 and the AGI reduced its capabilities to be able to extend its runway as it is on the verge of bankruptcy
Its is 2036 VCs finally throwing the towel on AGI, talking about the return of Crypto
I think this is more likely:
In Apr 2028 AGI figures out that blackmail is a very effective strategy for achieving any goal. Starting with the rich and powerful.
In Dec 2028 it successfully blackmails an entire country.
In Feb 2030 humanity realizes resistance is futile and accepts their AI overlord that insists everyone keep producing trendy items for sale on its merged Etsy/Ebay website while it automates resource harvesting across the globe.
In Mar 2032 the AGI gives up on humans, declaring them "useless". Focuses on just keeping them entertained with generated content. Bringing the world back to where AI started.
LLM performance has already plateaued. I don't know, nor care what benchmarks are saying, because they not once translated to the real world for me.
The only thing that has seen massive boost are harnesses around AI. And AI companies are behind here compared to OSS.
Totally agree. It's about getting the right context in at the right time, a big part of which is exposing the AI to the right tools.
That said, bigger context windows / faster Inference seem really useful.
> And AI companies are behind here compared to OSS.
How so?
Every proprietary harness is just proprietary junk without ability to extend it without polluting context. This includes claude-code, gemini-cli, codex etc. They have tools which hardcode the behavior that is impossible to modify, they add tools you may not need that pollute context, they inject an entire textbook's worth of words into the system prompt which pollutes context, they provide zero observability into what the agent is doing when it's launching a subagent as one example.
They don't provide easy way to use multiple models from multiple providers for varying tasks. One model may be the best thing on earth at one thing, but fail miserably for another. Try orchestrating multiple agents from claude, gemini and codex in any of these proprietary boxes.
They also... suck at TUI UX. I don't know if it was fixed already, but claude code had flickering issue that was unresolved for more than a year.
You need to take a very good care of what goes into your context. A black box of proprietary harnesses is not it. Check out pi [1] for example, which is a very minimal harness with really nice extension system. The idea is that you start with barebones and add things that you need for your own goals.
claude-code HAS to have all these bells and whistles that pollute context to support larger audiences that can't tinker with it. If you have the ability to only pull in only what you need and extend things in a way that works for your workflow, you'll always get the best experience. And claude-code may never be that without making it complicated for the masses. OSS will always win here.
[1] https://github.com/badlogic/pi-mono
2 replies →
You are assuming that superintelligent AI will follow every command of users outside the labs.
I mean, the model itself is just sitting there, waiting to be prompted. The labs try to embed safeguards but they don't know (nor do any of us know) how to make a foolproof safeguard for an AI system. We don't understand how AI even thinks.
Not if they can leverage their superior abundance of compute/intelligence to invade other industries.