Comment by butlike
6 days ago
Context is the plateau. It's why RAM prices are spiking. We're essentially throwing heap at the problem hoping it will improve. That's not engineering. It's not improving on a fundamental, technical level.
6 days ago
Context is the plateau. It's why RAM prices are spiking. We're essentially throwing heap at the problem hoping it will improve. That's not engineering. It's not improving on a fundamental, technical level.
> Context is the plateau. It's why RAM prices are spiking.
Yes, context is the plateau. But I don't think it the bottleneck is RAM. The mechanism described in "Attention is all you need" is O(N^2) where N is the size of the context window. I can "feel" this in everyday usage. As the context window size grows, the model responses slow down, a lot. That's due to compute being serialised because there aren't enough resources to do it in parallel. The resources are more likely compute and memory bandwidth than RAM.
If there is a breakthrough, I suspect it will be models turning the O(N^2) into O(N * ln(N)), which is generally how we speed things up in computer science. That in turn implies abstracting the knowledge in the context window into a hierarchical tree, so the attention mechanism only has to look across a single level in the tree. That in turn requires it to learn and memorise all these abstract concepts.
When models are trained the learn abstract concepts which they near effortlessly retrieve, but don't do that same type of learning when in use. I presume that's because it requires a huge amount of compute, repetition, and time. If only they could do what I do - go to sleep for 8 hours a day, and dream about the same events using local compute, and learn them. :D Maybe, one day, that will happen, but not any time soon.
If a bridge girder isn't strong enough to support a load, you add material in the right places to make a larger, stronger girder. That is engineering. The idea that if you're not making fundamental improvements to your formulation of steel you aren't progressing is absurd. If adding RAM leads to improvements, and we have the engineering ability to add more RAM, then we are still making progress.
Regardless of how true your statement is (just adding metal to a structure is commonly not a way to solve the problem you stated, it just makes the structure heavier which means other systems have more to support) the point is that it isnt exponential/fundamental progress, which is the type that would be needed to avoid the plateaus folks are mentioning. Also adding RAM doesnt give you even linear improvements, its logarithmic.
> just adding metal to a structure is commonly not a way to solve the problem you stated, it just makes the structure heavier which means other systems have more to support
As a mechanical engineer, that is exactly how you solve that problem.
> point is that it isnt exponential/fundamental progress
You just stuck the goalpost on a rocket and shot it into space. You'd be hard pressed to show evidence that progress in this field was ever exponential - in most fields it never was. Logarithmic progress is typical; you make a lot of progress early on picking the low hanging fruit figuring out the basics, and as the problems get harder and the theory better understood it takes more effort to make improvements, but fundamentally improvements continue.
Incremental progress from increasing scale is, again, perfectly cromulent. It's how we've made advanced computers that can fit in your pocket, it's how clothing became so cheap it's practically disposable, it's how you can fly across the country for less than the price of a nice dinner. Imagine looking at photolithography, textile manufacturing, or aircraft 5 years after they reached their modern forms and saying "this has plateaued".
2 replies →