Comment by elisharobinson
1 month ago
this is not only not true , it has no basis's in reality. In the "real" world there are tradeoff's and constraints. scaling does not work infinitely , and products which are delivered by good engneering cultures have a non linear growth ( bad vs very good ).
comming back to research one of the frontier model's deepseek was able to come close to SOTA with a relatively small budget because of one of their mixture of experts approach.
Yeah knowing that having infinite perfect data and infinite compute to train an end-to-end DNN is the way to solve anything is great, but not exactly helpful when you don't have either of those things. Not to mention having to actually deploy it on a low power system in the end.