Comment by gpjt
8 hours ago
But you can train a small LLM with a gaming graphics card -- I managed one on a GTX 1660. I don't think pg is suggesting that you try to chase the frontier. It's more like building your own OS in the 80s, or web server in the 90s -- sure, you'll never match the commercial offerings or the big OS projects, but building something from scratch within the limits of the hardware you can afford is amazing educationally.
"I'd build the foundation of knowledge to base a startup on later"
what kinds of startups ?
In my experience, having a solid understanding of the next level down in the stack -- the foundation you're building your startup on -- is really helpful. We built a PaaS, and knowing enough about Linux internals to be able to work out what would be easy and what would be hard meant that we could focus our efforts on high bang-for-buck features.
So I'd say that understanding LLMs to the level that you get to by training your own baby one would be a solid foundation for pretty much anything built on top of the "real" ones.