Comment by bitexploder
12 hours ago
But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)
12 hours ago
But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)
You can run it with 2x r9700 with 150-200 tokens with second. It is intelligent enough if you just point the docs / whatever for it.