Comment by LoveMistral
14 hours ago
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
Out of interest, have you tried the newer models? You are not describing my experience recently.
Yes - you are experiencing a mix of context caching and db retrieval from these mainstream model experiences.
Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code.
The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG).
On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.