Comment by PAndreew

I was thinking about this and there are several aspects that can still make this viable. 1) AI labs are incentivised to increase token consumption because literally that's their product. The only thing they sell AFIAK are tokens (and maybe a teensy bit of user data). So if you build a product that is actively reducing token consumption (which they simply cannot do without hurting themselves even if their marketing fluff says otherwise) you'll save large amounts of money for your customers and they'll choose you. 2) Big providers want to funnel every prompt into their servers. If you're in a regulated market or simply don't want to share every detail with an American or Chinese megacorp you are in trouble. BUT open weight models are now quite capable for "small business stuff" and they can be self hosted. If you can bundle this into your service, in other words actually care about their privacy, they will choose you. Even more so if you're in Europe.