Comment by inciampati
11 hours ago
There is no reason to believe that generalist models need to be transformer (or transformer hybrids) it's just that people are risk averse and not spending their resources on the many scaled risky bets needed to explore the possibilities. There is more there.
True. I'm a big fan of Cactus's work on the needle family of Simple Attention Network: https://github.com/cactus-compute/needle
That looks like a really nice and well put together library.
But it feels like it sort of backs up my point about there being good models at every size class. Fine tuning Needle looks automated. Yes, you need to know basics like what validation loss means and how to use Python, but otherwise it's all about creating the dataset.
Someone had to come up with needle in the first place though. And it's the kind of thing that's going to come from a big lab with an AGI monopoly ambition.