Comment by nijave

Forgot about the ol "but how many tokens did you spend to get _there_"--wish benchmarks would include the number of input/output tokens to achieve the score. I think the closest is Android Bench https://developer.android.com/bench although best you can do is extrapolate off time/cost (iirc they claim to prefer using provider's native API)

In general, smart models work fine with any tools, dumber models need better tools to achieve same results but better tools can eat more context/take more turns

I've gotten decent results with Llama 3.1 8b on Hugging Face tester with Exa MCP since it seems to dump sufficient context into WebSearch/WebFetch type calls even a crappy model almost always gets back what it needs as long as it calls the tool at least once. I had Claude Code look at previous sessions with SearXNG vibe MCP compared to Exa MCP and results got better when it modified SearXNG to work very similarly to Exa. Ended up with this https://gist.github.com/nijave/604c43e3e0fdcd60f5280d3a6b109... although it's really only optimized for "search" not "fetch" at this point. Fetch is basic niquests without Javascript or anything clever