Comment by EgregiousCube
2 days ago
Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls.
What kind of use case would be best for that shape?
2 days ago
Their published benchmarks seem to indicate that it's pretty good at coding and multimodal, but VERY good at successful tool calls.
What kind of use case would be best for that shape?
Debugging and diagnosis is very tool call heavy, whether that's grepping / transforming logs, calling out to profilers/tracers, or even just writing up incident reports.
Bug diagnostics is about being okay at coding but better at tooling.
Given a good diagnostic report, it can be handed to opus for the fix.
Opus is okay at writing reports, but it still regularly gets table widths wrong in typst documents, leaving the last column full of text but only a handful of characters wide.
I wonder if we'll start to see that pattern with every new release. Tool use likely changes rapidly, so the newest, rather than most intelligent, model may always have an edge.
What you mean.. The tools are all just invoking bash and terminal/cli cmds and http requests. Paradigms that have existed and stayed mostly unchanged for decades.
These do make up a huge % of tool calls, but I don't think these make up a huge % of tool call failures.
I see models fail on tool calls that involve API requests to a specific API, internal or cloned Makefile calls, npm run commands, etc.
Gemini 3.5 flash is better than fable at tool calling. Tool calling is probably one of the easier things to do post training for.
I don't use Gemini cli because they're so bad at agentic work/tool calling. I use their chatbot all the time, though.
This sounds... kind of useless? Really good JSON or similar constrained decoder performance is interesting, but normal decoder > tool validator loop with good error message > tool retry is almost always able to get a tool to work second try, and input is cached so it's not expensive.
Things are not always that simple, eg https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
The avg coding session has hundreds or thousands of tool calls. Even a 5% failure rate noticeably notches up token use and cost. See Gemini.
Yes, but each tool call has a different failure %. The tool calls that make up the majority of volume like grep are going to have nowhere near a 5% failure. A custom user-defined skill having a 5% failure rate is probably fine.