Comment by viccis
15 hours ago
I'll try to get something set up to try this out. I've been working on an ESP32 based Echo replacement that sends audio back to a backend server I run, and one question I had was whether models small enough to run on a Mac Mini or even smaller hardware are good enough to handle basic tool calling functionality with a bit of reasoning where needed.
I have a test suite that tries like ~36 different scenarios, including things like starting multiple timers, saying "actually cancel that timer" and whether it knows to do that one you just created. Basic decision making on top of tool calling. I found so far that, for example, Qwen3.8 on my local machine does pretty poorly even relative to Gemma4 E4B (~9.6gb) and that the best price/performance outcome I've found so far with openrouter is actually GPT Luna, but obviously I'd love to get something that works as well running locally for privacy reasons.
Would love to try this out, I'll just need to tweak my benchmarker to use however this serves it.
Hey there! If you end up trying out needle on the test suite it would be very useful for us if you could share some failure modes of the model! We are always trying to understand where the model isn't doing good and where we can make it better.
For your question on tool calling, I think you will find that the model is pretty good at simpler tool calls and parallel ones, but can struggle with implied references and multistep reasoning. These are definitely things that can improve with task-specific finetuning but for some things you just have to have a model that is properly sized. That said, we are always trying to improve the model so that it can handle an ever larger set of queries
Thanks for responding. I ran it just now and it looks like it could be useful if I change and limit the scope of the kinds of actions I need. Here is a breakdown of where it struggled vs some local models (~8-12B parameters running on a 16GB Mac Mini or my desktop's 3080), bear in mind I had Claude integrate it into the benchmarks and this is its interpretation, not my own:
What it gets wrong:
- It copies numbers instead of converting them. "25 minute timer" becomes duration_seconds: 25, and "twelve minutes" becomes 120. The first one comes with 100% confidence.
- It picks the wrong action. "take the paper towels off the list" became an add. "remind me in 20 minutes" became a timer. "add five minutes to the pasta timer" became a new timer plus a cancel.
- It never declined anything with our full tool set. Background chatter became note_save "blue one" at 0.99 confidence. "play some jazz" became a screen card, and "wake me up at 6 30" became a 630-second timer.
- It can't use household context. Notes, timer names and reminder IDs have no place in its input. Passing them anyway made results worse (5 of 27 single-turn requests right, versus 8 of 27 without), so the backend now leaves them out.
- Follow-ups mostly broke. "take off the last one" removed the whole list.
This is extremely useful feedback for us, thanks! I think the easiest thing here that can be fixed with tool definitions is the number conversions. Additionally, the model tends to work better with fewer tools. We will definitely be focusing on better context usage and followups going forward as well.