Comment by JKCalhoun
21 days ago
"I actively avoid cloud based LLMs… This means I don't really have a good grasp of SOTA LLM performance or accuracy…
…but then all of your metrics and data seem to focus 100% on accuracy. You need to address speed."
I wonder, if you were to use cloud-based LLMs more often, you might find that accuracy (fidelity?) is indeed more more lacking in your local models.
You can always just throw hardware at your speed problems after all.
I agree accuracy isn't maybe the best word here, I used it as it was used in the original post, mainly a as a catchall for "everything but speed", so fidelity, perplexity, etc.
I also agree that if I spent more time using cloud based LLMs, I would very much find local LLMs less capable and useful. Comparison is the thief of joy though, and I'd rather feel blissfully ignorance towards SOTA LLMs rather than a dependence on them.
Before taking a local focus approach, LLMs increasingly left me feeling a mixture of FOMO, sadness and futility towards the future of software and tech. I assume it's 100% a me problem, but it has it's benefits:)
No, I'm a fan of local as well. For me though, there is just such a fascination that I can have something like this sitting on my own hard drive. It's okay that it's not a "frontier model".