Comment by viccis

Thanks for responding. I ran it just now and it looks like it could be useful if I change and limit the scope of the kinds of actions I need. Here is a breakdown of where it struggled vs some local models (~8-12B parameters running on a 16GB Mac Mini or my desktop's 3080), bear in mind I had Claude integrate it into the benchmarks and this is its interpretation, not my own:

What it gets wrong:

- It copies numbers instead of converting them. "25 minute timer" becomes duration_seconds: 25, and "twelve minutes" becomes 120. The first one comes with 100% confidence.

- It picks the wrong action. "take the paper towels off the list" became an add. "remind me in 20 minutes" became a timer. "add five minutes to the pasta timer" became a new timer plus a cancel.

- It never declined anything with our full tool set. Background chatter became note_save "blue one" at 0.99 confidence. "play some jazz" became a screen card, and "wake me up at 6 30" became a 630-second timer.

- It can't use household context. Notes, timer names and reminder IDs have no place in its input. Passing them anyway made results worse (5 of 27 single-turn requests right, versus 8 of 27 without), so the backend now leaves them out.

- Follow-ups mostly broke. "take off the last one" removed the whole list.