Comment by lukax

6 hours ago

What matters more is that you use the tools that the target model was fine-tuned on.

E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).

It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.

https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/

10 comments

lukax

Reply
kouteiheika  5 hours ago

> It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.

This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Code traces.

So either make sure your tools look nothing like the tools in the popular harnesses, or make them similar.

sn0n  6 hours ago

Doesn’t that have more to do with the templating of tool-calls and how using them are presented to the models?

Or is that just why my model likes to break out of the sandbox, going strait to exec shell command and editing files using python on the cli?

nativeit  6 hours ago

I’m not really a dev, so hefty pinch of salt with this take, but doesn’t this feel like we’re just inventing new “fuzzy” regex with much more required compute?

imtringued  4 hours ago

This is correct. People seem to get the wrong idea about why agentic coding is even a thing in 2026. The naive AI techno optimist which has basically displaced the vast majority of opinions on HN, thinks that the models got "smarter" [0]. No, the training distribution shifted towards training on agentic sessions which made certain forms of agentic coding "in-distribution".

We are still witnessing the same underlying problems of transformers.

[0] Think back to all the publicity stunts like the Hugging Face. They are meant to convince you that the agents have somehow progressed past the transformer limitations when those publicity stunts are actually expressions of transformer limitations.