Strongly recommend this blog post too which is a much more detailed and persuasive version of the same point. The author actually goes and builds a coding agent from zero: https://ampcode.com/how-to-build-an-agent
It is indeed astonishing how well a loop with an LLM that can call tools works for all kinds of tasks now. Yes, sometimes they go off the rails, there is the problem of getting that last 10% of reliability, etc. etc., but if you're not at least a little bit amazed then I urge you go to and hack together something like this yourself, which will take you about 30 minutes. It's possible to have a sense of wonder about these things without giving up your healthy skepticism of whether AI is actually going to be effective for this or that use case.
This "unreasonable effectiveness" of putting the LLM in a loop also accounts for the enormous proliferation of coding agents out there now: Claude Code, Windsurf, Cursor, Cline, Copilot, Aider, Codex... and a ton of also-rans; as one HN poster put it the other day, it seems like everyone and their mother is writing one. The reason is that there is no secret sauce and 95% of the magic is in the LLM itself and how it's been fine-tuned to do tool calls. One of the lead developers of Claude Code candidly admits this in a recent interview.[0] Of course, a ton of work goes into making these tools work well, but ultimately they all have the same simple core.
Can't think of anything an LLM is good enough at to let them do on their own in a loop for more than a few iterations before I need to reign it back in.
The main problem with agents is that they aren't reflecting on their own performance and pausing their own execution to ask a human for help aggressively enough. Agents can run on for 20+ iterations in many cases successfully, but also will need hand holding after every iteration in some cases.
They're a lot like a human in that regard, but we haven't been building that reflection and self awareness into them so far, so it's like a junior that doesn't realize when they're over their depth and should get help.
I built android-use[1] using LLM. It is pretty good at self healing due to the "loop", it constantly checks if the current step is actually a progress or regress and then determines next step. And the thing is nothing is explicitly coded, just a nudge in the prompts.
There's also this one which uses pocketflow, a graph abstraction library to create something similar [0]. I've been using it myself and love the simplicity of it.
Today I tried "vibe-coding" for the first time using GPT-4o and 4.1. I did it manually - just feeding compilation errors, warnings, and suggestions in a loop via the canvas interface. The file was small, around 150 lines.
It didn't go well. I started with 4o:
- It used a deprecated package.
- After I pointed that out, it didn't update all usages - so I had to fix them manually.
- When I suggested a small logic change, it completely broke the syntax (we're talking "foo() } return )))" kind of broken) and never recovered. I gave it the raw compilation errors over and over again, but it didn't even register the syntax was off - just rewrote random parts of the code instead.
- Then I thought, "maybe 4.1 will be better at coding" (as advertized). But 4.1 refused to use the canvas at all. It just explained what I could change - as in, you go make the edits.
- After some pushing, I got it to use the canvas and return the full code. Except it didn't - it gave me a truncated version of the code with comments like "// omitted for brevity".
That's when I gave up.
Do agents somehow fix this? Because as it stands, the experience feels completely broken. I can't imagine giving this access to bash, sounds way too dangerous.
If you prompt an LLM with the general concept that it made a mistake, it will make more of them, because that is the plausible continuation of the conversation, and they are trained first and foremost as plausibility engines.
If you'll allow an exaggeration to illustrate, imagine a conversation with an idiot. You tell them they made a mistake. Now which is more plausible: they continue being an idiot, or they have an amazingly sudden gain in competence? The latter is simply less plausible. This is even without considering the dilution of attention following input token inflation that modern LLM architectures can only partially mitigate with context optimisation, chunking, compression etc.
The central dogma of all these models is that plausibility will (with continuing architectural research and sheer scale) converge towards correctness, or "faithfulness" as it's sometimes called in the literature. This hypothesis remains very far from proven. Nevertheless, it follows that you are much better off starting a new prompt with directives that guard against the original error without labelling it as such, rather than continuing a prompt chain that is gradually degrading because you have literally prompted the model to be less competent.
Even if you're not up on the state-of-the-art of their internals, this is what mechanical sympathy looks like when iterating on results with LLMs.
I've switched to o4-mini-high via ChatGPT as my default model for a lot of code because it can use its search function to lookup the latest documentation.
You can tell it "look up the most recent version of library X and use that" and it will often work!
I even used it for a frustrating upgrade recently - I pasted in some previous code and prompted this:
This code needs to be upgraded to the new recommended JavaScript library from Google. Figure out what that is and then look up enough documentation to port this code to it.
>That's because models have training cut-off dates
When I pointed out that it used a deprecated package, it agreed and even cited the correct version after which it was deprecated (way back in 2021). So it knows it's deprecated, but the next-token prediction (without reasoning or tools) still can't connect the dots when much of the training data (before 2021) uses that package as if it's still acceptable.
>I've switched to o4-mini-high via ChatGPT as my default model for a lot of code because it can use its search function to lookup the latest documentation.
GPT 4.1 and 4o score very low on the Aider coding benchmark. You only start to get acceptable results with models that score 70%+ in my experience. Even then, don't expect it to do anything complex without a lot of hand-holding. You start to get a sense for what works and what doesn't.
I get that it's frustrating to be told "skill issue," but using an LLM is absolutely a skill and there's a combination of understanding the strengths of various tools, experimenting with them to understand the techniques, and just pure practice.
I think if I were giving access to bash, though, it would definitely be in a docker container for me as well.
Except the skill involved is believing in random people's advice that a different model will surely be better with no fundamental reason or justification as to why. The benchmarks are not applicable when trying to apply the models to new work and benchmarks by there nature do not describe suitability to any particular problem.
The LLM is a problem solver but not a repository of documentation. Neural networks are not designed for that. They model at a conceptual level. It still needs to look up specific API documentation like human developers.
You could use o3 and ask it to search the web for documentation and read that first, but it's not efficient. The professional LLM coding assistant tools manage the context properly.
Eh, given how much about anything these models know without googling, they are certainly knowledge repositories, designed for it or not. How deep and up-to-date their knowledge of some obscure subject, is another question.
The other day I used the Cline plugin for VSCode with Claude to create an Android app prototype from "scratch", i.e. starting from the usual template given to you by Android Studio. It produced several thousand lines of code, there was not a single compilation error, and the app ended up doing exactly what I wanted – modulo a bug or two, which were caused not by the LLM's stupidity but by weird undocumented behavior of the rather arcane Android API in question. (Which is exactly why I wanted a quick prototype.)
After pointing out the bugs to the LLM, it successfully debugged them (with my help/feedback, i.e. I provided the output of the debug messages it had added to the code) and ultimately fixed them. The only downside was that I wasn't quite happy with the quality of the fixes – they were more like dirty hacks –, but oh well, after another round or two of feedback we got there, too. I'm sure one could solve that more generally, by putting the agent writing the code in a loop with some other code reviewing agent.
I can be frustrating at times. but my experience is the more you try the better you become at knowing what to ask and to expect.
But I guess you understand now why some people say vibe coding is a bit overrated: https://www.lycee.ai/blog/why-vibe-coding-is-overrated
> Today I tried "vibe-coding" for the first time using GPT-4o and 4.1. I did it manually
You set yourself up to fail from the get go. But understandable. If you don't have a lot of experience in this space, you will struggle with low quality tools and incorrect processes. But, if you stick with it, you will discover better tools and better processes.
My best results are usually with 4o-mini-high, o3 is sometimes pretty good
I personally don’t like the canvas. I prefer the output on the chat
And a lot of times I say: provide full code for this file, or provide drop-in replacement (when I don’t want to deal with all the diffs). But usually at around 300-400 lines of code, it starts getting bad and then I need to refactor to break stuff up into multiple files (unless I can focus on just one method inside a file)
o3 is shockingly good actually. I can’t use it often due to rate limiting, so I save it for the odd occasion. Today I asked it how I could integrate a tree of Swift binary packages within an SDK, and detect internal version clashes, and it gave a very well researched and sensible overview. And gave me a new idea that I‘ll try.
You should try Cursor or Windsurf, with Claude or Gemini model. Create a documentation file first. Generate tests for everything. The more the better. Then let it cycle 100 times until tests pass.
Normal programming is like walking, deliberate and sure. Vibe coding is like surfing, you can't control everything, just hit yes on auto. Trust the process, let it make mistakes and recover on its own.
Given that analogy, surely you could understand why someone would much rather walk than surf to their destination? Especially people who are experienced marathon runners.
I find that writing a thorough design spec is really worth it. Also, asking for its reaction. "What's missing?" "Should I do X or Y" does good things for its thought process, like engaging a younger programmer in the process.
Definitely, I ask for a plan and then, even if it's obvious, I ask questions and discuss it. I also point it as samples of code that I like with instructions for what is good about it.
Once we have settled on a plan, I ask it to break it into phases that can be tested (I am not one for a unit testing) to lock in progress. Claude LOVES that. It organizes a new plan and, at the end of each phase tells me how to test (curl, command line, whatever is appropriate) and what I should see that represents success.
The most important thing I have figured out is that Claude is a collaborator, not a minion. I agree with visarga, it's much more like surfing that walking. Also, Trust... but Verify.
GPT 4o and 4.1 are both pretty terrible for coding to be honest, try Sonnet 3.7 in Cline (VSCode extension).
LLMs don't have up to date knowledge of packages by themselves that's a bit like buying a book and expecting it to have up to date world knowledge, you need to supplement / connect it to a data source (e.g. web search, documentation and package version search etc.).
It's completely broken for me over 400 lines (Claude 3.7, paid Cursor)
The worst is when I ask something complex, the model generates 300 lines of good code and then timeouts or crashes. If I ask to continue it will mess up the code for good, eg. starts generating duplicated code or functions which don't match the rest of the code.
In this case, sorry to say but it sounds like there's a tooling issue, possibly also a skill issue. Of course you can just use the raw ChatGPT web interface but unless you seriously tune its system/user prompt, it's not going to match what good tooling (which sets custom prompts) will get you. Which is kind of counter-intuitive. A paragraph or three fed in as the system prompt is enough to influence behavior/performance so significantly? It turns out with LLMs the answer is yes.
I've been using Claude Code, ie, a terminal interface to Sonnet 3.7 since the day it came out in mid March. I have done substantial CLI apps, full stack web systems and a ton of utility crap. I am much more ambitious because of it, much as I was in the past when I was running a programming team.
I'm sure it is much the same as this under the hood though Anthropic has added many insanely useful features.
Nothing is perfect. Producing good code requires about the same effort as it did when I was running said team. It is possible to get complicated things working and find oneself in a mess where adding the next feature is really problematic. As I have learned to drive it, I have to do much less remediation and refactoring. That will never go away.
I cannot imagine what happened to poor kgeist. I have had Claude make choices I wouldn't and do some stupid stuff, never enough that I would even think about giving up on it. Almost always, it does a decent job and, for a most stuff, the amount of work it takes off of my brain is IMMENSE.
And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool.
And, just for fun, I opened it in a non-code archive directory. It was a junk drawer that I've been filling for thirty years. "What's in this directory?" "Read the old resumes and write a new one." "What are my children's names?" Also amazing.
Recently I had to define a remote data structure, specify the API to request it, implement parsing and storage, and show it to the user.
Claude was able to handle all of these tasks simultaneously, so I could see how small changes at either end would impact the intermediate layers. I iterated on many ideas until I settled on the best overall solution for my use case.
Being able to iterate like that through several layers of complexity was eye-opening. It made me more productive while giving me a better understanding of how the different components fit together.
> And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool.
Yeah this is literally just so enjoyable. Stuff that would be an up-hill battle to get included in a sprint takes 5 minutes. It makes it feel like a whole team is just sitting there, waiting to eagerly do my bidding with none of the headache waiting for work to be justified, scheduled, scoped, done, and don't even have to justify rejecting it if I don't like the results.
This morning I used cursor to extract a few complex parts of my game prototype's "main loop", and then generate a suite of tests for those parts. In total I have 341 tests written by Cursor covering all the core math and other components.
It has been a bit like herding cats sometimes, it will run away with a bad idea real fast, but the more constraints I give it telling it what to use, where to put it, giving it a file for a template, telling it what not to do, the better the results I get.
In total it's given me 3500 lines of test code that I didn't need to write, don't need to fix, and can delete and regenerate if underlying assumptions change. It's also helped tune difficulty curves, generate mission variations and more.
Writing tests, in my experience, is by far the best use case for LLMs. It eliminates hours or days of drudgery and toil, and can provide coverage of so many more edge cases that I can think of myself. Plus it makes your code more robust! It’s just wonderful all around.
Yeah I generated another 60 tests / 500 lines of test code since I posted my comment. I must have written 100,000s of lines of test codes the hard way, it's not just faster than me it's better than me when it comes to the coverage, and it requires less of my help than a number of developers I've had under me.
I'm very excited about tool use for LLMs at the moment.
The trick isn't new - I first encountered it with the ReAcT paper two years ago - https://til.simonwillison.net/llms/python-react-pattern - and it's since been used for ChatGPT plugins, and recently for MCP, and all of the models have been trained with tool use / function calls in mind.
What's interesting today is how GOOD the models have got at it. o3/o4-mini's amazing search performance is all down to tool calling. Even Qwen3 4B (2.6GB from Ollama, runs happily on my Mac) can do tool calling reasonably well now.
I gave a workshop at PyCon US yesterday about building software on top of LLMs - https://simonwillison.net/2025/May/15/building-on-llms/ - and used that as an excuse to finally add tool usage to an alpha version of my LLM command-line tool. Here's the section of the workshop that covered that:
My LLM package can now reliably count the Rs in strawberry as a shell one-liner:
llm --functions '
def count_char_in_string(char: str, string: str) -> int:
"""Count the number of times a character appears in a string."""
return string.lower().count(char.lower())
' 'Count the number of Rs in the word strawberry' --td
Unfortunately, I haven't had a good experience with tool use. I built a simple agent that can access my calendar and add events, and GPT-4.1 regularly gaslights me, saying "I've added this to your calendar" when it hasn't. It takes a lot of prompting for it to actually add the event, but it will insist it has, even though it never called the tool.
Does anyone know of a fix? I'm using the OpenAI agents SDK.
> "Oh, this test doesn't pass... let's just skip it," it sometimes says, maddeningly.
Here is a wild idea. Imagine running a companion, policy-enforcing LLM, independently and in parallel, which is given instructions to keep the main LLM behaving according to instructions.
If the companion LLM could - in real time - ban the coding LLM from emitting "let's just skip it" by seeing the tokens "let's just" and then biasing the output such that the word "skip" becomes impossible to emit.
Banning the word "skip" from following "let's just", forces the LLM down a new path away from the undesired behavior.
It's like Structured Outputs or JSON mode, but driven by a companion LLM, and dynamically modified in real time as tokens are emitted.
If the idea works, you could prompt the companion LLM to do more advanced stuff - eg. ban a coding LLM from making tests pass by deleting the test code, ban it from emitting pointless comments... all the policies that we put into system prompts today and pray the LLM will do, would go into the companion LLM's prompt instead.
Along these lines, if the main LLM goes down a bad path, you could _rewind_ the model to before it started going down the bad path -- the watcher LLM doesn't necessarily have to guess that "skip" is a bad token after the words "let's just", it could instead see "let's just skip the test" and go "nope, rolling back to the token "just " and rerolling with logit_bias={"skip":-10,"omit":-10,"hack":-10}".
Of course doing that limits which model providers you can work with (notably, OpenAI has gotten quite hostile to power users doing stuff like that over the past year or so).
Kind of seems an optimization: if the “token ban” is a tool call, you can see that being too slow to run for every token. Provided rewinding is feasible, your idea could make it performant enough to be practical.
If it works to run a second LLM to check the first LLM, then why couldn't a "mixture of experts" LLM dedicate one of its experts to checking the results of the others? Or why couldn't a test-time compute "thinking" model run a separate thinking thread that verifies its own output? And if that gets you 60% of the way there, then there could be yet another thinking thread that verifies the verifier, etc.
I didn't know of that paper, and thought the title was a riff on Karpathy's Unreasonable Effectiveness of RNNs in 2015[1]. Even if my thinking is correct, as it very well might be given the connection RNNs->LLMs, Karpathy might have himself made his title a play on Wigner's (though he doesn't say so).
This is great, and I like seeing all the implementations people are making for themselves.
Anyone using any opensource tooling that bundles this effectively to allow different local models to be used in this fashion?
I am thinking this would be nice to run fully locally to access my code or my private github repos from my commandline and switch models out (assuming through llama.ccp or Ollama)?
The blog post in question is on the site for Sketch, which appears to use Docker containers. That said, I use Claude Code, which just uses unsandboxed commands with manual approval.
What's your concern? An accident or an attacker? For accidents, I use git and backups and develop in a devcontainer. For an attacker, bash just seems like an ineffective attack vector; I would be more worried about instructing the agent to write a reverse shell directly into the code.
I've also been defining agents as "LLM call in a while loop with tools" to my co-workers as well — I'd add that if you provide it something like a slack tool, you can enable the LLM to ask for help (human in the (while) loop).
Started with a math visualizer for machine learning, saw an HN post for this soon after and scrapped it. It was better done by someone else.
Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else.
It is like every single original notion I have is immediately done by someone else at the exact same time.
I think I will just move on to rudimentary systems programming stuff and avoid creative and original thinking, just need basic and low profile employment.
> Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else.
If it helps, "TFA" was not the originator here and is merely simplifying concepts from fairly established implementations in the wild. As simonw mentions elsewhere, it goes back to at least the ReAct paper and maybe even more if you consider things like retrieval-augmented generation.
I've found sonnet-3.7 to be incredibly inconsistent. It can do very well but has a strong tendency to get off-track and run off and do weird things.
3.5 is better for this, ime. I hooked claude desktop up to an MCP server to fake claude-code less the extortionate pricing and it works decently. I've been trying to apply it for rust work; it's not great yet (still doesn't really seem to "understand" rust's concepts) but can do some stuff if you make it `cargo check` after each change and stop it if it doesn't.
I expect something like o3-high is the best out there (aider leaderboards support this) either alone or in combination with 4.1, but tbh that's out of my price range. And frankly, I can't mentally get past paying a very high price for an LLM response that may or may not be useful; it leaves me incredibly resentful as a customer that your model can fail the task, requiring multiple "re-rolls", and you're passing that marginal cost to me.
I am avoiding the cost of API access by using the chat/ui instead, in my case Google Gemini 2.5 Pro with the high token window. Repomix a whole repo. Paste it in with a standard prompt saying "return full source" (it tends to not follow this instruction after a few back and forths) and then apply the result back on top of the repo (vibe coded https://github.com/radekstepan/apply-llm-changes to help me with that).
Else yeah, $5 spent on Cline with Claude 3.7 and instead of fixing my tests, I end up with if/else statements in the source code to make the tests pass.
I decided to experiment with Claude Code this month. The other day it decided the best way to fix the spec was to add a conditional to the test that causes it to return true before getting to the thing that was actually supposed to be tested.
I’m finding it useful for really tedious stuff like doing complex, multi step terminal operations. For the coding… it’s not been great.
Cool tool. What format does it expect from the model?
I’ve been looking for something that can take “bare diffs” (unified diffs without line numbers), from the clipboard and then apply them directly on a buffer (an open file in vscode)
None of the paste diff extension for vscode work, as they expect a full unified diff/patch
I also tried a google-developed patch tool, but also wasn’t very good at taking in the bare diffs, and def couldn’t do clipboard
I've been using Mistral Medium 3 last couple of days, and I'm honestly surprised at how good it is. Highly recommend giving it a try if you haven't, especially if you are trying to reduce costs. I've basically switched from Claude to Mistral and honestly prefer it even if costs were equal.
I seem to be alone in this but the only methods truly good at coding are slow heavy test time compute models.
o1-pro and o1-preview are the only models I've ever used that can reliably update and work with 1000 LOC without error.
I don't let o3 write any code unless it's very small. Any "cheap" model will hallucinate or fail massively when pushed.
One good tip I've done lately. Remove all comments in your code before passing or using LLMs, don't let LLM generated comments persist under any circumstance.
Interesting. I've never tested o1-pro because it's insanely expensive but preview seemed to do okay.
I wouldn't be shocked if huge, expensive-to-run models performed better and if all the "optimized" versions were actually labs trying to ram cheaper bullshit down everyone's throat. Basically chinesium for LLMs; you can afford them but it's not worth it. I remember someone saying o1 was, what, 200B dense? I might be misremembering.
I built this very same thing today! The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically.
I used ollama to build this and ollama supports tool calling natively, by passing a `tools=[...]` in the Python SDK. The tools can be regular Python functions with docstrings that describe the tool use. The SDK handles converting the docstrings into a format the LLM can recognize, so my tool's code documentation becomes the model's source of truth. I can also include usage examples right in the docstring to guide the LLM to work closely with all my available tools. No system prompt needed!
Moreover, I wrote all my tools in a separate module, and just use `inspect.getmembers` to construct the `tools` list that i pass to Ollama. So when I need to write a new tool, I just write another function in the tools module and it Just Works™
Paired with qwen 32b running locally, i was fairly satisfied with the output.
> The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically.
It looks like this one does that too.
msg = [ handle_tool_call(tc) for tc in tool_calls ]
Not only can this be an effective strategy for coding tasks, but it can also be used for data querying. Picture a text-to-SQL agent that can query database schemas, construct queries, run explain plans, inspect the error outputs, and then loop several times to refine. That's the basic architecture behind a tool I built, and I have been amazed at how well it works. There have been multiple times when I've thought, "Surely it couldn't handle THIS prompt," but it does!
This is the same principle that we use at Brokk for Search and for Architect. (https://brokk.ai/)
The biggest caveat: some models just suck at tool calling, even "smart" models like o3. I only really recommend Gemini Pro 2.5 for Architect (smart + good tool calls); Search doesn't require as high a degree of intelligence and lots of models work (Sonnet 3.7, gpt-4.1, Grok 3 are all fine).
I'm curious about your experiences with Gemini Pro 2.5 tool calling. I have tried using it in agent loops (in fact, sketch has some rudimentary support I added), and compared with the Anthropic models I have had to actively reprompt Gemini regularly to make tool calls. Do you consider it equivalent to Sonnet 3.7? Has it required some prompt engineering?
Confession time: litellm still doesn't support parallel tool calls with Gemini models [https://github.com/BerriAI/litellm/issues/9686] so we wrote our own "parallel tool calls" on top of Structured Output. It did take a few iterations on the prompt design but all of it was "yeah I can see why that was ambiguous" kinds of things, no real complaints.
GP2.5 does have a different flavor than S3.7 but it's hard to say that one is better or worse than the other [edit: at tool calling -- GP2.5 is definitely smarter in general]. GP2.5 is I would say a bit more aggressive at doing "speculative" tool execution in parallel with the architect, e.g. spawning multiple search agent calls at the same time, which for Brokk is generally a good thing but I could see use cases where you'd want to dial that back.
That's really cool.
One week ago I implemented an SQL tool and it works really well. But sometimes it still just makes up table/column names. Luckily it can read the error and correct itself.
But today I went to the next level.
I gave the LLM two tools. One web search tool and one REST tool.
I told it at what URL it can find API docs. Then I asked it to perform some tasks for me.
It was really cool to watch an AI read docs, make api calls and try again (REPL) until it worked
This is for running locally in the terminal (or in WSL2 if you are on Windows).
Here is an example session from a similar script that I <cough>ChatGPT</cough> wrote in Perl, illustrating the same concept:
$ ~/agentloop.pl
YOUR PROMPT:
Is Python installed on this system? If so, which version?
BASH STDIN:
python --version 2>&1 || python3 --version 2>&1
BASH STDOUT:
/bin/bash: python: command not found
Python 3.6.7
LLM RESPONSE:
Python is installed on this system. The version available is Python 3.6.7. However, it looks like the python executable (python) is not directly available, but python3 (or a direct version path) exists and resolves to Python 3.6.7.
If you need to use Python, you should use python3 as the command.
Has anybody written an Electron app yet which injects tools into existing chat UIs, letting you you expose whatever you want from your machine to them via Node? I was planning to create something like that for the BigCo internal chat UI I work on.
Strongly recommend this blog post too which is a much more detailed and persuasive version of the same point. The author actually goes and builds a coding agent from zero: https://ampcode.com/how-to-build-an-agent
It is indeed astonishing how well a loop with an LLM that can call tools works for all kinds of tasks now. Yes, sometimes they go off the rails, there is the problem of getting that last 10% of reliability, etc. etc., but if you're not at least a little bit amazed then I urge you go to and hack together something like this yourself, which will take you about 30 minutes. It's possible to have a sense of wonder about these things without giving up your healthy skepticism of whether AI is actually going to be effective for this or that use case.
This "unreasonable effectiveness" of putting the LLM in a loop also accounts for the enormous proliferation of coding agents out there now: Claude Code, Windsurf, Cursor, Cline, Copilot, Aider, Codex... and a ton of also-rans; as one HN poster put it the other day, it seems like everyone and their mother is writing one. The reason is that there is no secret sauce and 95% of the magic is in the LLM itself and how it's been fine-tuned to do tool calls. One of the lead developers of Claude Code candidly admits this in a recent interview.[0] Of course, a ton of work goes into making these tools work well, but ultimately they all have the same simple core.
[0] https://www.youtube.com/watch?v=zDmW5hJPsvQ
Can't think of anything an LLM is good enough at to let them do on their own in a loop for more than a few iterations before I need to reign it back in.
They're extremely good at burning through budgets, and get even better when unattended
The main problem with agents is that they aren't reflecting on their own performance and pausing their own execution to ask a human for help aggressively enough. Agents can run on for 20+ iterations in many cases successfully, but also will need hand holding after every iteration in some cases.
They're a lot like a human in that regard, but we haven't been building that reflection and self awareness into them so far, so it's like a junior that doesn't realize when they're over their depth and should get help.
I built android-use[1] using LLM. It is pretty good at self healing due to the "loop", it constantly checks if the current step is actually a progress or regress and then determines next step. And the thing is nothing is explicitly coded, just a nudge in the prompts.
1. clickclickclick - A framework to let local LLMs control your android phone (https://github.com/BandarLabs/clickclickclick)
Ah, it’s Thorsten Ball!
I thoroughly enjoyed his “writing an interpreter”. I guess I’m going to build an agent now.
There's also this one which uses pocketflow, a graph abstraction library to create something similar [0]. I've been using it myself and love the simplicity of it.
[0] https://github.com/The-Pocket/PocketFlow-Tutorial-Cursor/blo...
Should we change the link above to use `?utm_source=hn&utm_medium=browser` before opening it?
fixed :)
thanks for the rec. and yeah agreed with the observations as well
For "that last 10% of reliability" RL is actually working pretty well right now too! https://openpipe.ai/blog/art-e-mail-agent
Today I tried "vibe-coding" for the first time using GPT-4o and 4.1. I did it manually - just feeding compilation errors, warnings, and suggestions in a loop via the canvas interface. The file was small, around 150 lines.
It didn't go well. I started with 4o:
- It used a deprecated package.
- After I pointed that out, it didn't update all usages - so I had to fix them manually.
- When I suggested a small logic change, it completely broke the syntax (we're talking "foo() } return )))" kind of broken) and never recovered. I gave it the raw compilation errors over and over again, but it didn't even register the syntax was off - just rewrote random parts of the code instead.
- Then I thought, "maybe 4.1 will be better at coding" (as advertized). But 4.1 refused to use the canvas at all. It just explained what I could change - as in, you go make the edits.
- After some pushing, I got it to use the canvas and return the full code. Except it didn't - it gave me a truncated version of the code with comments like "// omitted for brevity".
That's when I gave up.
Do agents somehow fix this? Because as it stands, the experience feels completely broken. I can't imagine giving this access to bash, sounds way too dangerous.
If you prompt an LLM with the general concept that it made a mistake, it will make more of them, because that is the plausible continuation of the conversation, and they are trained first and foremost as plausibility engines.
If you'll allow an exaggeration to illustrate, imagine a conversation with an idiot. You tell them they made a mistake. Now which is more plausible: they continue being an idiot, or they have an amazingly sudden gain in competence? The latter is simply less plausible. This is even without considering the dilution of attention following input token inflation that modern LLM architectures can only partially mitigate with context optimisation, chunking, compression etc.
The central dogma of all these models is that plausibility will (with continuing architectural research and sheer scale) converge towards correctness, or "faithfulness" as it's sometimes called in the literature. This hypothesis remains very far from proven. Nevertheless, it follows that you are much better off starting a new prompt with directives that guard against the original error without labelling it as such, rather than continuing a prompt chain that is gradually degrading because you have literally prompted the model to be less competent.
Even if you're not up on the state-of-the-art of their internals, this is what mechanical sympathy looks like when iterating on results with LLMs.
"It used a deprecated package"
That's because models have training cut-off dates. It's important to take those into account when working with them: https://simonwillison.net/2025/Mar/11/using-llms-for-code/#a...
I've switched to o4-mini-high via ChatGPT as my default model for a lot of code because it can use its search function to lookup the latest documentation.
You can tell it "look up the most recent version of library X and use that" and it will often work!
I even used it for a frustrating upgrade recently - I pasted in some previous code and prompted this:
This code needs to be upgraded to the new recommended JavaScript library from Google. Figure out what that is and then look up enough documentation to port this code to it.
It did exactly what I asked: https://simonwillison.net/2025/Apr/21/ai-assisted-search/#la...
>That's because models have training cut-off dates
When I pointed out that it used a deprecated package, it agreed and even cited the correct version after which it was deprecated (way back in 2021). So it knows it's deprecated, but the next-token prediction (without reasoning or tools) still can't connect the dots when much of the training data (before 2021) uses that package as if it's still acceptable.
>I've switched to o4-mini-high via ChatGPT as my default model for a lot of code because it can use its search function to lookup the latest documentation.
Thanks for the tip!
3 replies →
GPT 4.1 and 4o score very low on the Aider coding benchmark. You only start to get acceptable results with models that score 70%+ in my experience. Even then, don't expect it to do anything complex without a lot of hand-holding. You start to get a sense for what works and what doesn't.
https://aider.chat/docs/leaderboards/
That been said, Claude Sonnet 3.7 seems to do very well at a recursive approach to writing a program whereas other models don't fare as well.
I get that it's frustrating to be told "skill issue," but using an LLM is absolutely a skill and there's a combination of understanding the strengths of various tools, experimenting with them to understand the techniques, and just pure practice.
I think if I were giving access to bash, though, it would definitely be in a docker container for me as well.
Sure, you can probably get better at it, but is it really worth the effort over just getting better at programming?
5 replies →
Except the skill involved is believing in random people's advice that a different model will surely be better with no fundamental reason or justification as to why. The benchmarks are not applicable when trying to apply the models to new work and benchmarks by there nature do not describe suitability to any particular problem.
The default chat interface is the wrong tool for the job.
The LLM needs context.
https://github.com/marv1nnnnn/llm-min.txt
The LLM is a problem solver but not a repository of documentation. Neural networks are not designed for that. They model at a conceptual level. It still needs to look up specific API documentation like human developers.
You could use o3 and ask it to search the web for documentation and read that first, but it's not efficient. The professional LLM coding assistant tools manage the context properly.
Eh, given how much about anything these models know without googling, they are certainly knowledge repositories, designed for it or not. How deep and up-to-date their knowledge of some obscure subject, is another question.
1 reply →
As others have noted, you sound about 3 months behind the leading edge. What you describe is like my experience from February.
Switch to Claude (IMSHO, I think Gemini is considered on par). Use a proper coding tool, cutting & pasting from the chat window is so last week.
Instead of churning on frontend frameworks while procrastinating about building things we've moved onto churning dev setups for micro gains.
The other day I used the Cline plugin for VSCode with Claude to create an Android app prototype from "scratch", i.e. starting from the usual template given to you by Android Studio. It produced several thousand lines of code, there was not a single compilation error, and the app ended up doing exactly what I wanted – modulo a bug or two, which were caused not by the LLM's stupidity but by weird undocumented behavior of the rather arcane Android API in question. (Which is exactly why I wanted a quick prototype.)
After pointing out the bugs to the LLM, it successfully debugged them (with my help/feedback, i.e. I provided the output of the debug messages it had added to the code) and ultimately fixed them. The only downside was that I wasn't quite happy with the quality of the fixes – they were more like dirty hacks –, but oh well, after another round or two of feedback we got there, too. I'm sure one could solve that more generally, by putting the agent writing the code in a loop with some other code reviewing agent.
> I'm sure one could solve that more generally, by putting the agent writing the code in a loop with some other code reviewing agent.
This x 100. I get so much better quality code if I have LLMs review each other's code and apply corrections. It is ridiculously effective.
1 reply →
I can be frustrating at times. but my experience is the more you try the better you become at knowing what to ask and to expect. But I guess you understand now why some people say vibe coding is a bit overrated: https://www.lycee.ai/blog/why-vibe-coding-is-overrated
"Overrated" is one way to call it.
Giving sharp knives to monkeys would be another.
GPT4o and 4.1 are definitely not the best models to use here. Use Claude 3.5/3.7, Gemini Pro 2.5 or o3. All of them work really well for small files.
> Today I tried "vibe-coding" for the first time using GPT-4o and 4.1. I did it manually
You set yourself up to fail from the get go. But understandable. If you don't have a lot of experience in this space, you will struggle with low quality tools and incorrect processes. But, if you stick with it, you will discover better tools and better processes.
4o and 4.1 are not very good at coding
My best results are usually with 4o-mini-high, o3 is sometimes pretty good
I personally don’t like the canvas. I prefer the output on the chat
And a lot of times I say: provide full code for this file, or provide drop-in replacement (when I don’t want to deal with all the diffs). But usually at around 300-400 lines of code, it starts getting bad and then I need to refactor to break stuff up into multiple files (unless I can focus on just one method inside a file)
o3 is shockingly good actually. I can’t use it often due to rate limiting, so I save it for the odd occasion. Today I asked it how I could integrate a tree of Swift binary packages within an SDK, and detect internal version clashes, and it gave a very well researched and sensible overview. And gave me a new idea that I‘ll try.
2 replies →
Drop in replacement files per update should be done on the heavy test time compute methods.
o1-pro, o1-preview can generate updated full file responses into the 1k LOC range.
It's something about their internal verification methods that make it an actual viable development method.
1 reply →
Aider's benchmarks show 4.1 (and 4o) work better in its architect mode, for planning the changes, and o3 for making the actual edits
You have that backwards. The leaderboard results have the thinking model as the architect.
In this case, o3 is the architect and 4.1 is the editor.
You should try Cursor or Windsurf, with Claude or Gemini model. Create a documentation file first. Generate tests for everything. The more the better. Then let it cycle 100 times until tests pass.
Normal programming is like walking, deliberate and sure. Vibe coding is like surfing, you can't control everything, just hit yes on auto. Trust the process, let it make mistakes and recover on its own.
Given that analogy, surely you could understand why someone would much rather walk than surf to their destination? Especially people who are experienced marathon runners.
6 replies →
I find that writing a thorough design spec is really worth it. Also, asking for its reaction. "What's missing?" "Should I do X or Y" does good things for its thought process, like engaging a younger programmer in the process.
Definitely, I ask for a plan and then, even if it's obvious, I ask questions and discuss it. I also point it as samples of code that I like with instructions for what is good about it.
Once we have settled on a plan, I ask it to break it into phases that can be tested (I am not one for a unit testing) to lock in progress. Claude LOVES that. It organizes a new plan and, at the end of each phase tells me how to test (curl, command line, whatever is appropriate) and what I should see that represents success.
The most important thing I have figured out is that Claude is a collaborator, not a minion. I agree with visarga, it's much more like surfing that walking. Also, Trust... but Verify.
This is a great time to be a programmer.
GPT 4o and 4.1 are both pretty terrible for coding to be honest, try Sonnet 3.7 in Cline (VSCode extension).
LLMs don't have up to date knowledge of packages by themselves that's a bit like buying a book and expecting it to have up to date world knowledge, you need to supplement / connect it to a data source (e.g. web search, documentation and package version search etc.).
Agents definitely fix this. When you can run commands and edit files, the agent can test its code by itself and fix any issues.
150 lines? I find can quickly scale to around 1500 lines, and then start more precision on the classes, and functions I am looking to modify
It's completely broken for me over 400 lines (Claude 3.7, paid Cursor)
The worst is when I ask something complex, the model generates 300 lines of good code and then timeouts or crashes. If I ask to continue it will mess up the code for good, eg. starts generating duplicated code or functions which don't match the rest of the code.
4 replies →
You gotta use a reasoning model.
Use Claude Sonnet with an IDE.
Try o3 please. Via UI.
In this case, sorry to say but it sounds like there's a tooling issue, possibly also a skill issue. Of course you can just use the raw ChatGPT web interface but unless you seriously tune its system/user prompt, it's not going to match what good tooling (which sets custom prompts) will get you. Which is kind of counter-intuitive. A paragraph or three fed in as the system prompt is enough to influence behavior/performance so significantly? It turns out with LLMs the answer is yes.
skill issue.
The fact that you're using 4o and 4.1 rather than claude is already a huge mistake in itself.
> Because as it stands, the experience feels completely broken
Broken for you. Not for everyone else.
I've been using Claude Code, ie, a terminal interface to Sonnet 3.7 since the day it came out in mid March. I have done substantial CLI apps, full stack web systems and a ton of utility crap. I am much more ambitious because of it, much as I was in the past when I was running a programming team.
I'm sure it is much the same as this under the hood though Anthropic has added many insanely useful features.
Nothing is perfect. Producing good code requires about the same effort as it did when I was running said team. It is possible to get complicated things working and find oneself in a mess where adding the next feature is really problematic. As I have learned to drive it, I have to do much less remediation and refactoring. That will never go away.
I cannot imagine what happened to poor kgeist. I have had Claude make choices I wouldn't and do some stupid stuff, never enough that I would even think about giving up on it. Almost always, it does a decent job and, for a most stuff, the amount of work it takes off of my brain is IMMENSE.
And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool.
And, just for fun, I opened it in a non-code archive directory. It was a junk drawer that I've been filling for thirty years. "What's in this directory?" "Read the old resumes and write a new one." "What are my children's names?" Also amazing.
And this is still early days. I am so happy.
Recently I had to define a remote data structure, specify the API to request it, implement parsing and storage, and show it to the user.
Claude was able to handle all of these tasks simultaneously, so I could see how small changes at either end would impact the intermediate layers. I iterated on many ideas until I settled on the best overall solution for my use case.
Being able to iterate like that through several layers of complexity was eye-opening. It made me more productive while giving me a better understanding of how the different components fit together.
> And, for good measure, it does a wonderful job of refactoring. Periodically, I have a session where I look at the code, decide how it could be better and instruct Claude. Huge amounts of complexity, done. "Change this data structure", done. It's amazingly cool.
Yeah this is literally just so enjoyable. Stuff that would be an up-hill battle to get included in a sprint takes 5 minutes. It makes it feel like a whole team is just sitting there, waiting to eagerly do my bidding with none of the headache waiting for work to be justified, scheduled, scoped, done, and don't even have to justify rejecting it if I don't like the results.
This morning I used cursor to extract a few complex parts of my game prototype's "main loop", and then generate a suite of tests for those parts. In total I have 341 tests written by Cursor covering all the core math and other components.
It has been a bit like herding cats sometimes, it will run away with a bad idea real fast, but the more constraints I give it telling it what to use, where to put it, giving it a file for a template, telling it what not to do, the better the results I get.
In total it's given me 3500 lines of test code that I didn't need to write, don't need to fix, and can delete and regenerate if underlying assumptions change. It's also helped tune difficulty curves, generate mission variations and more.
Writing tests, in my experience, is by far the best use case for LLMs. It eliminates hours or days of drudgery and toil, and can provide coverage of so many more edge cases that I can think of myself. Plus it makes your code more robust! It’s just wonderful all around.
Yeah I generated another 60 tests / 500 lines of test code since I posted my comment. I must have written 100,000s of lines of test codes the hard way, it's not just faster than me it's better than me when it comes to the coverage, and it requires less of my help than a number of developers I've had under me.
I'm very excited about tool use for LLMs at the moment.
The trick isn't new - I first encountered it with the ReAcT paper two years ago - https://til.simonwillison.net/llms/python-react-pattern - and it's since been used for ChatGPT plugins, and recently for MCP, and all of the models have been trained with tool use / function calls in mind.
What's interesting today is how GOOD the models have got at it. o3/o4-mini's amazing search performance is all down to tool calling. Even Qwen3 4B (2.6GB from Ollama, runs happily on my Mac) can do tool calling reasonably well now.
I gave a workshop at PyCon US yesterday about building software on top of LLMs - https://simonwillison.net/2025/May/15/building-on-llms/ - and used that as an excuse to finally add tool usage to an alpha version of my LLM command-line tool. Here's the section of the workshop that covered that:
https://building-with-llms-pycon-2025.readthedocs.io/en/late...
My LLM package can now reliably count the Rs in strawberry as a shell one-liner:
Was the workshop recorded?
No video or audio, just my handouts.
I love the odd combination of silliness and power in this.
Sharing an agent framework (from scratch) that works very well with just 7 composable tools: read, write, diff, browse, command, think and ask.
Just pushed an update this week for OpenAI-compatibility too!
https://github.com/aperoc/toolkami
Unfortunately, I haven't had a good experience with tool use. I built a simple agent that can access my calendar and add events, and GPT-4.1 regularly gaslights me, saying "I've added this to your calendar" when it hasn't. It takes a lot of prompting for it to actually add the event, but it will insist it has, even though it never called the tool.
Does anyone know of a fix? I'm using the OpenAI agents SDK.
> "Oh, this test doesn't pass... let's just skip it," it sometimes says, maddeningly.
Here is a wild idea. Imagine running a companion, policy-enforcing LLM, independently and in parallel, which is given instructions to keep the main LLM behaving according to instructions.
If the companion LLM could - in real time - ban the coding LLM from emitting "let's just skip it" by seeing the tokens "let's just" and then biasing the output such that the word "skip" becomes impossible to emit.
Banning the word "skip" from following "let's just", forces the LLM down a new path away from the undesired behavior.
It's like Structured Outputs or JSON mode, but driven by a companion LLM, and dynamically modified in real time as tokens are emitted.
If the idea works, you could prompt the companion LLM to do more advanced stuff - eg. ban a coding LLM from making tests pass by deleting the test code, ban it from emitting pointless comments... all the policies that we put into system prompts today and pray the LLM will do, would go into the companion LLM's prompt instead.
Wonder what the Outlines folks think of this!
Along these lines, if the main LLM goes down a bad path, you could _rewind_ the model to before it started going down the bad path -- the watcher LLM doesn't necessarily have to guess that "skip" is a bad token after the words "let's just", it could instead see "let's just skip the test" and go "nope, rolling back to the token "just " and rerolling with logit_bias={"skip":-10,"omit":-10,"hack":-10}".
Of course doing that limits which model providers you can work with (notably, OpenAI has gotten quite hostile to power users doing stuff like that over the past year or so).
That’s a really neat idea.
Kind of seems an optimization: if the “token ban” is a tool call, you can see that being too slow to run for every token. Provided rewinding is feasible, your idea could make it performant enough to be practical.
If it works to run a second LLM to check the first LLM, then why couldn't a "mixture of experts" LLM dedicate one of its experts to checking the results of the others? Or why couldn't a test-time compute "thinking" model run a separate thinking thread that verifies its own output? And if that gets you 60% of the way there, then there could be yet another thinking thread that verifies the verifier, etc.
Because if the agent and governor are trained together, the shared reward function will corrupt the governor.
Assuming the title is a play on the paper "The Unreasonable Effectiveness of Mathematics in the Natural Sciences"[1][2] by Eugene Wigner.
[1]: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
[2]: https://www.hep.upenn.edu/~johnda/Papers/wignerUnreasonableE...
That may be its primogenitor, but it's long since become a meme: https://scholar.google.com/scholar?q=unreasonable+effectiven...
I didn't know of that paper, and thought the title was a riff on Karpathy's Unreasonable Effectiveness of RNNs in 2015[1]. Even if my thinking is correct, as it very well might be given the connection RNNs->LLMs, Karpathy might have himself made his title a play on Wigner's (though he doesn't say so).
[1] https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Unreasonable effectiveness of [blah] has been a thing for decades if not centuries. It's not new.
1 reply →
> If you don't have some tool installed, it'll install it.
Terrifying. LLMs are very 'accommodating' and all they need is someone asking them to do something. This is like SQL injection, but worse.
This is great, and I like seeing all the implementations people are making for themselves.
Anyone using any opensource tooling that bundles this effectively to allow different local models to be used in this fashion?
I am thinking this would be nice to run fully locally to access my code or my private github repos from my commandline and switch models out (assuming through llama.ccp or Ollama)?
What protection do people use when enabling an LLM to run `bash` on your machine ? Do you run it in a Docker container / LXC boundary ? `chroot` ?
The blog post in question is on the site for Sketch, which appears to use Docker containers. That said, I use Claude Code, which just uses unsandboxed commands with manual approval.
What's your concern? An accident or an attacker? For accidents, I use git and backups and develop in a devcontainer. For an attacker, bash just seems like an ineffective attack vector; I would be more worried about instructing the agent to write a reverse shell directly into the code.
I've also been defining agents as "LLM call in a while loop with tools" to my co-workers as well — I'd add that if you provide it something like a slack tool, you can enable the LLM to ask for help (human in the (while) loop).
Woke up this morning to start on a new project.
Started with a math visualizer for machine learning, saw an HN post for this soon after and scrapped it. It was better done by someone else.
Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else.
It is like every single original notion I have is immediately done by someone else at the exact same time.
I think I will just move on to rudimentary systems programming stuff and avoid creative and original thinking, just need basic and low profile employment.
Everything has been done before, sometimes before you were even born. The trick is to just not google it first and do it anyways.
What is your motive for creating things? Does it really matter if there is competition?
> Started on an LLM app that looped outputs, saw this post soon after and scrapped it. It was better done by someone else.
If it helps, "TFA" was not the originator here and is merely simplifying concepts from fairly established implementations in the wild. As simonw mentions elsewhere, it goes back to at least the ReAct paper and maybe even more if you consider things like retrieval-augmented generation.
I've found sonnet-3.7 to be incredibly inconsistent. It can do very well but has a strong tendency to get off-track and run off and do weird things.
3.5 is better for this, ime. I hooked claude desktop up to an MCP server to fake claude-code less the extortionate pricing and it works decently. I've been trying to apply it for rust work; it's not great yet (still doesn't really seem to "understand" rust's concepts) but can do some stuff if you make it `cargo check` after each change and stop it if it doesn't.
I expect something like o3-high is the best out there (aider leaderboards support this) either alone or in combination with 4.1, but tbh that's out of my price range. And frankly, I can't mentally get past paying a very high price for an LLM response that may or may not be useful; it leaves me incredibly resentful as a customer that your model can fail the task, requiring multiple "re-rolls", and you're passing that marginal cost to me.
I am avoiding the cost of API access by using the chat/ui instead, in my case Google Gemini 2.5 Pro with the high token window. Repomix a whole repo. Paste it in with a standard prompt saying "return full source" (it tends to not follow this instruction after a few back and forths) and then apply the result back on top of the repo (vibe coded https://github.com/radekstepan/apply-llm-changes to help me with that). Else yeah, $5 spent on Cline with Claude 3.7 and instead of fixing my tests, I end up with if/else statements in the source code to make the tests pass.
I decided to experiment with Claude Code this month. The other day it decided the best way to fix the spec was to add a conditional to the test that causes it to return true before getting to the thing that was actually supposed to be tested.
I’m finding it useful for really tedious stuff like doing complex, multi step terminal operations. For the coding… it’s not been great.
2 replies →
Cool tool. What format does it expect from the model?
I’ve been looking for something that can take “bare diffs” (unified diffs without line numbers), from the clipboard and then apply them directly on a buffer (an open file in vscode)
None of the paste diff extension for vscode work, as they expect a full unified diff/patch
I also tried a google-developed patch tool, but also wasn’t very good at taking in the bare diffs, and def couldn’t do clipboard
2 replies →
Guess it was trained by scraping thedailywtf.com
I've been using Mistral Medium 3 last couple of days, and I'm honestly surprised at how good it is. Highly recommend giving it a try if you haven't, especially if you are trying to reduce costs. I've basically switched from Claude to Mistral and honestly prefer it even if costs were equal.
How are you running the model? Mistral’s api or some local version through ollama, or something else?
3 replies →
I seem to be alone in this but the only methods truly good at coding are slow heavy test time compute models.
o1-pro and o1-preview are the only models I've ever used that can reliably update and work with 1000 LOC without error.
I don't let o3 write any code unless it's very small. Any "cheap" model will hallucinate or fail massively when pushed.
One good tip I've done lately. Remove all comments in your code before passing or using LLMs, don't let LLM generated comments persist under any circumstance.
Interesting. I've never tested o1-pro because it's insanely expensive but preview seemed to do okay.
I wouldn't be shocked if huge, expensive-to-run models performed better and if all the "optimized" versions were actually labs trying to ram cheaper bullshit down everyone's throat. Basically chinesium for LLMs; you can afford them but it's not worth it. I remember someone saying o1 was, what, 200B dense? I might be misremembering.
1 reply →
If you're interested in hacking on agent loops, come join us in the OpenHands community!
Here's our (slightly more complicated) agent loop: https://github.com/All-Hands-AI/OpenHands/blob/f7cb2d0f64666...
I built this very same thing today! The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically.
I used ollama to build this and ollama supports tool calling natively, by passing a `tools=[...]` in the Python SDK. The tools can be regular Python functions with docstrings that describe the tool use. The SDK handles converting the docstrings into a format the LLM can recognize, so my tool's code documentation becomes the model's source of truth. I can also include usage examples right in the docstring to guide the LLM to work closely with all my available tools. No system prompt needed!
Moreover, I wrote all my tools in a separate module, and just use `inspect.getmembers` to construct the `tools` list that i pass to Ollama. So when I need to write a new tool, I just write another function in the tools module and it Just Works™
Paired with qwen 32b running locally, i was fairly satisfied with the output.
> The only difference is that i pushed the tool call outputs into the conversation history and resent it back to the LLM for it to summarize, or perform further tool calls, if necessary, automagically.
It looks like this one does that too.
Ah, failed to notice that.
I was so excited because this was exactly what I coded up today, I jumped straight to the comments.
Yes, agent loops are simple, except, as the article says, a bit of "pump and circumstance"!
If anyone is interested, I tried to put together a minimal library (no dependency) for TypeScript: https://github.com/hbbio/nanoagent
Not only can this be an effective strategy for coding tasks, but it can also be used for data querying. Picture a text-to-SQL agent that can query database schemas, construct queries, run explain plans, inspect the error outputs, and then loop several times to refine. That's the basic architecture behind a tool I built, and I have been amazed at how well it works. There have been multiple times when I've thought, "Surely it couldn't handle THIS prompt," but it does!
Here's an AWS post that goes into detail about this approach: https://aws.amazon.com/blogs/machine-learning/build-a-robust...
Link to the full script is broken?
Yes!
Han Xiao at Jina wrote a great article that goes into a lot more detail on how to turn this into a production quality agentic search: https://jina.ai/news/a-practical-guide-to-implementing-deeps...
This is the same principle that we use at Brokk for Search and for Architect. (https://brokk.ai/)
The biggest caveat: some models just suck at tool calling, even "smart" models like o3. I only really recommend Gemini Pro 2.5 for Architect (smart + good tool calls); Search doesn't require as high a degree of intelligence and lots of models work (Sonnet 3.7, gpt-4.1, Grok 3 are all fine).
I'm curious about your experiences with Gemini Pro 2.5 tool calling. I have tried using it in agent loops (in fact, sketch has some rudimentary support I added), and compared with the Anthropic models I have had to actively reprompt Gemini regularly to make tool calls. Do you consider it equivalent to Sonnet 3.7? Has it required some prompt engineering?
Confession time: litellm still doesn't support parallel tool calls with Gemini models [https://github.com/BerriAI/litellm/issues/9686] so we wrote our own "parallel tool calls" on top of Structured Output. It did take a few iterations on the prompt design but all of it was "yeah I can see why that was ambiguous" kinds of things, no real complaints.
GP2.5 does have a different flavor than S3.7 but it's hard to say that one is better or worse than the other [edit: at tool calling -- GP2.5 is definitely smarter in general]. GP2.5 is I would say a bit more aggressive at doing "speculative" tool execution in parallel with the architect, e.g. spawning multiple search agent calls at the same time, which for Brokk is generally a good thing but I could see use cases where you'd want to dial that back.
That's really cool. One week ago I implemented an SQL tool and it works really well. But sometimes it still just makes up table/column names. Luckily it can read the error and correct itself.
But today I went to the next level. I gave the LLM two tools. One web search tool and one REST tool.
I told it at what URL it can find API docs. Then I asked it to perform some tasks for me.
It was really cool to watch an AI read docs, make api calls and try again (REPL) until it worked
I found this out too - it is quite easy and effective:
https://benhouston3d.com/blog/building-an-agentic-code-from-...
Huh, isn't this already built-in, in most chat UIs?
This is for running locally in the terminal (or in WSL2 if you are on Windows).
Here is an example session from a similar script that I <cough>ChatGPT</cough> wrote in Perl, illustrating the same concept:
$ ~/agentloop.pl
YOUR PROMPT:
Is Python installed on this system? If so, which version?
BASH STDIN:
python --version 2>&1 || python3 --version 2>&1
BASH STDOUT:
/bin/bash: python: command not found
Python 3.6.7
LLM RESPONSE:
Python is installed on this system. The version available is Python 3.6.7. However, it looks like the python executable (python) is not directly available, but python3 (or a direct version path) exists and resolves to Python 3.6.7.
If you need to use Python, you should use python3 as the command.
Which chat UI lets me use my local tools? Like git, find, pnpm, curl...?
Has anybody written an Electron app yet which injects tools into existing chat UIs, letting you you expose whatever you want from your machine to them via Node? I was planning to create something like that for the BigCo internal chat UI I work on.
[dead]
[flagged]