Comment by LoveMistral
15 hours ago
Same. Mistral 7b has been more than I ever needed for text for years now.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
Sounds interesting. Can you give more details on your workflow and what tasks you use it for?
Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work.
Actually built a full invoicing product for that, using it too.
I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).
Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
Out of interest, have you tried the newer models? You are not describing my experience recently.
1 reply →
What are some limitations you have found with using a smaller model like that?
As a developer, none related to the model. It’s faster, lighter, makes less assumptions.
For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia.
But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know.
For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users.
For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings).
For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.