Comment by aunty_helen

19 hours ago

Thanks so much. 27b is a favourite model and last night my dev ran some benchmarks to achieve 133tks about 3.2x which changes the economics.

2 comments

aunty_helen

Reply

xenonite 18 hours ago

great to hear, which quant + KV quant + KV size are you on? is it llama.cpp?

aunty_helen 6 hours ago

Here's the writeup from the experiments that claude did: https://claude.ai/artifact/X6WJVtKdL1LrhYuCMqdXvG#b9a4d3e2-8...