Comment by CapsAdmin
3 months ago
I work on this typed lua language in lua, and sometimes use llms to help fix internal analyzer stuff, which works 30% of the time for complex, and sometimes not at all, but helps me find a solution in the end.
However when I ask an llm to generate my typed lua code, with examples and all, on how the syntax is supposed to be, it mostly gets it wrong.
my syntax for tables/objects is: local x: {foo = boolean}
but an llm will most likely gloss over this and always use : instead of = local x: {foo: boolean}
I've had success in the past with getting it to write YueScript/Moonscript (which is not a very large part of its training data) by pointing it to the root URL for the language docs and thus making that part of the context.
If your typed version of Lua has a syntax checker, you could also have it try to use that first on any code it's generated
Are you using a coding agent or just an llm chat interface? Do you have a linter or compiler that will catch the misuse that you’ve hooked up to the agent?
I've dabbled with claude code in this particular project, but not much. My short experience with it is that it's slow, costly and goes off the rails easily.
I prefer to work with more isolated parts of the code. But again, I don't really know all that much about agents.
One thing I wanted to do on my project is reorganize all the tests, which sounds like an agent job. But I'd imagine I need to define some hard programmatic constraints to make sure tests are not lost or changed in the process.
Agents aren’t magic. They are loops with tool calls in them that help keep agents on track. And most of the agent systems have some manner of hook that you can put your own tools in to enforce things like types and styles.
I’ve had good experiences writing small scripts and linters to enforce things that agents get wrong frequently. What’s nice about those is that the agents are very good at writing them and they are easy to verify. Plus they are valuable for new humans devs as well.