Comment by charlie-83
1 day ago
Out if interest, what kind of codebases are you able to get AI to do these things on. Everytime I have tried it with even simpler things than these it has failed spectacularly. Every example I see of people doing this kind of thing seems to be on some kind if web development so I have a hypothesis that AI might currently be much worse for the kinds of codebases I work on.
I currently work for a finance-related scaleup. So backend systems, with significant challenges related to domain complexity and scalability, but nothing super low level either.
It does take a bit to understand how to prompt in a way that the results are useful, can you share what you tried so far?
I have tried on a lot of different projects.
I have a codebase in Zig and it doesn't understand Zig at all.
I have another which is embedded C using zephyr RTOS. It doesn't understand zephyr at all and even if it could, it can't read the documentation for the different sensors nor can it plug in cables.
I have a tui project in rust using ratatui. The core of the project is dealing with binary files and the time it takes to explain to it how specific bits of data are organised in the file and then check it got everything perfectly correct (it never has) is more than the time to just write the code. I expect I could have more success on the actual TUI side of things but haven't tried too much since I am trying to learn rust with this project.
I just started an android app with flutter/dart. I get the feeling it will work well for this but I am yet to verify since I need to learn enough flutter to be able to judge it
My dayjob is a big C++ codebase making a GUI app with Qt. The core of it is all dealing with USB devices and Bluetooth protocols which it doesn't understand at all. We also have lots of very complicated C++ data structures, I had hoped that the AI would be able to at least explain them to me but it just makes stuff up everytime. This also means that getting it to edit any part of the codebase touching this sort if thing doesn't work. It just rips up any thread safety or allocates memory incorrectly etc. It also doesn't understand the compiler errors at all, I had a circular dependency and tried to get it to solve it but I had to give so many clues I basically told it what the problem was.
I really expected it to work very well for the Qt interface since building UI is what everyone seems to be doing with it. But the amount of hand holding it requires is insane. Each prompt feels like a monkey's paw. In every experiment I've done it would have been faster to just write it myself. I need to try getting it to write an entirely new pice of UI from scratch since I've only been editing existing UI so far.
Some of this is clearly a skill issue since I do feel myself getting better at prompting it and getting better results. However, I really do get the feeling that it either doesn't work or doesn't work as well on my code bases as other ones.
> I have a codebase in Zig and it doesn't understand Zig at all.
> I have another which is embedded C using zephyr RTOS. It doesn't understand zephyr at all and even if it could, it can't read the documentation for the different sensors nor can it plug in cables.
If you use Cursor, you can let it index the documentation for whatever language or framework you want [0], and it works exceptionally well. Don't rely solely on the LLM's training data, allow it to use external resources. I've done that and it solves many of the issues you're talking about.
[0] https://docs.cursor.com/context/@-symbols/@-docs
1 reply →
I work in Python, Swift, and Objective-C. AI tools work great in all of these environment. It's not just limited to web development.
I suppose saying that I've only seen it in web development is a bit of an exaggeration. It would be more accurate to say that I haven't seen any examples of people using AI on a codebase that looks like on of the ones I work on. Clearly I am biased just lump all the types of coding I'm not interested in into "web development"
It's not about backend or frontend, it's mostly about languages. If you're using niche stuff or languages that don't have an online or digital presence, the LLMs will be confused. Stuff like assembler or low-level C aren't really in their vocabulary.
C#, Go, Python for example work perfectly well and you can kinda test the LLMs preference by asking them to write a program to solve a problem, but don't specify the language.
That’s my experience too. It also fails terribly with ElasticSearch probably because the documentation doesn’t have a lot of examples. ChatGPT, copilot and claude were all useless for that and gave completely plausible nonsense. I’ve used it with most success for writing unit tests and definitely shell scripts.