Comment by hirvi74
12 hours ago
"It" is a lot of things.
Languages:
- C#/.NET: Sufficient sometimes, but not how I'd write things. Most results at least compile, but I have noticed plenty of defiance towards particular instructions, e.g, "Do not use <x>, use <y>" -> code contains <x> and not <y>.
- C#/Godot: I have noticed the greatest amounts of defiance here. Not to mention most results are an 80/20 implementation of what I asked for. And no, I am not trying to one-shot a full game or anything.
- AArch64 and x86: great results surprisingly, though only small amounts were produced. Mainly, assistance with RE-ing and cracking some binaries from https://crackmes.one or where ever.
- The Lord's Language (Swift): Maybe the LLMs are better at SwiftUI/Swift, but I have had some rough results going down the opposite direction of the software stack. I have on/off been working on a personal, FOSS "productivity" tool for macOS, e.g., mouseless navigation, window management, GUI automation, etc.. This type of development requires a significant amount work with C APIs like CoreGraphics, Accessibility, CoreFoundation, etc.. The code isn't the problem for me, it's the lack of useful debugging. LLMs, last I have tried (around Opus 4.6 times), seemed to really struggle with things like CoreGraphics Y-axis coordinates being inverted compared AppKit's and other stuff like that.
- Applescript (GUI automation): Do not even waste your time trying (I fault no LLMs for this either).
- elisp: the code is usually sufficient, though package config can be a little dicey.
- Shell scripts (Zsh, Bash, Powershell): great results.
- Python: I try to avoid this language unless necessary, but the results have been great.
These days, I use the plain Web chat interfaces for about 95% of my usage compared to the CLI harnesses. Sometime ago, I realized I get better results that way. With the web chat, I would say my results have been outstanding.
Direct feedback:
You have to give up on style. "not how I'd write things" is not a blocker. Defiance of instructions is normal, you just have to steer it and correct. There's no substitute for diligence yet.
80/20 - this means your scope was too large, split the scope or tell the agent to revert, split the scope, and try again.
RE and assembler: it's really good at this stuff. It can patch almost any binary with the right tools
Swift: you have to give it tool usage in whatever result you're wanting. If it's a macos app, you have to let the LLM pilot it to get feedback, or build an extensive end to end test suite that it can drive autonomously. If you get into the loop on changes it'll feel awful and like no time savings. Review at the level of using the app and looking at the code, not in process or reviewing every tool call or diff.
applescript: works great, I have a bunch of automation set up this way, what problems are you seeing?
elisp: tough language, llms kinda hate parentheses unless you're really tight on the linting, and elisp is enough of its own animal that the training for e.g. common lisp isn't great.
python: will suck unless you enable all the typechecking, make it use bdd, and have a linter/formatter run precommit and yell at the robot for you.
Web vs cli: you should use the cli 100% because it lets you change the environment, if you're getting better results on web, you haven't set your local environment up very well. My personal preference is to run my own dev server on aws but that's spendy.