Comment by cadamsdotcom
8 hours ago
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
But this is exactly what the AI labs should be doing ...
Love LLMs gonna keep using them. It feels like your suggested approach is expensive, in terms of tokens. I feel (second time I say this) that when I steer the process I get pretty good results vs my coworkers that let the LLMs run away. I do have data on our token usage, not much in terms of quality of the delivery.
I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
Thr thing is that both you and your agent should have a way to verify the solution.
OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
> expensive, in terms of tokens.
No amount of tokens can come close to my hourly rate.
Do you steer your agents by manually running every single test and linter and reporting the results back to them?