Comment by socketcluster
8 days ago
1000 tokens per second. Crazy. I'm wondering what this leads to.
Imagine the massive amount of software that's going to get built. It will be like reinventing the wheel in a million ways. There will be thousands of alternative internet ecosystems to choose from and each one of then would offer every software system, platform and application that one could possibly need; fully compatible with data transferrable across any application within the same ecosystem. Some ecosystems would facilitate data transfers in and out. Ecosystems would be competing against each other; all different, but ultimately yielding very similar results. The competitive edge of one ecosystem over another would be purely grounded in narrative with no basis in reality because the differences between the best ecosystems would be meaningless. That said there would also be bad ecosystems where a lot of people may get trapped. Some people would get lost in the junk.
It's cool but TPS count is not a meaningful limiting factor to new software. These small models are also too dumb for QA in complex codebases (for now), but on a future timeline they are super cool. Model distillation and ablation generally is very interesting.
I predict this comment will feel very 640k-is-enough in a few years. And by years I mean in 2 weeks.