Comment by OGWhales
6 hours ago
I've not found it that great at programming in cobol, at least in comparison to its ability with other languages it seems to be noticeably worse, though we aren't using any models that were specifically trained on cobol. It is still useful for doing simple and tedious tasks, for example constructing a file layout based on info I fed it can be a time saver, otherwise I feel it's pretty limited by the necessary system specifics and really large context window needed to understand what is actually going on in these systems. I do really like being able to feed it a whole manual and let it act as a sort of advanced find. Working in a mainframe environment often requires looking for some obscure info, typically in a large PDF that's not always easy to find what you need, so this is pretty nice.
AI isn’t particularly great with C, Zig, or Rust either in my experience. It can certainly help with snippets of code and elucidate complex bitwise mathematics, and I’ll use it for those tedious tasks. And it’s a great research assistant, helping with referencing documentation. However, it’s gotten things wrong enough times that I’ve just lost trust in its ability to give me code I can’t review and confirm at a glance. Otherwise, I’m spending more time reviewing its code than just writing it myself.
AI is pretty bad at Python and Go as well. It depends a lot on who uses it though. We have a lot of non-developers who make things work with Python. A lot of it will never need a developer because it being bad doesn't matter for what it does. Some of it needs to be basically rewritten from scratch.
Over all I think it's fine.
I do love AI for writing yaml and bicep. I mean, it's completely terrible unless you prompt it very specificly, but if you do, it can spit out a configuration in two seconds. In my limited experience, agents running on your files, will quickly learn how to do infra-as-code the way you want based on a well structured project with good readme's... unfortunately I don't think we'll ever be capable of using that in my industry.
If it's bad at python the most popular language what language it's good at? If you see the other comments they're basically mentioning most programming languages
7 replies →
> AI is pretty bad at Python and Go as well.
It great in Golang IF its one shot tasks. LLMs seem to degrade a lot when they are forced to work on existing code bases (even their own). What seems to be more a issue with context sizes growing out of control way too fast (and this is what degrades LLMs the most).
So far Opus 4.5 has been the one LLM that keeps mostly coding in a, how to say, predictable way even with a existing code base. It requires scaffolding and being very clear with your coding requests. But not like the older models where they go off script way too much or rewrite code in their own style.
For me Opus 4.5 has reached that sweet spot of productivity and not just playing around with LLMs and undoing mistakes.
The problem with LLMs is a lot of times a mix of LLM issues, people giving different requests, context overload, different models doing better with different languages, the amount of data it needs to alter etc... This makes the results very mixed from one person to another, and harder to quantify.
Even the different in a task makes the difference between a person one day glorifying a LLM and a few weeks later complaining it was nerfed, when it was not. Just people doing different work / different prompts and ...
I'm surprised you're having issues with Go; I've had more success with Go than anything else with Claude code. Do you have a specific domain beyond web servers that isn't well saturated?
with all those languages listed in this thread,it explains why I don't trust or use AI when I code.
That's basically all the languages that I am using...
For the AI fans in here, what languages are you using? Typescript only would be my guess?
8 replies →
I’ve found claide code to be amazing at go. This is all nuts because experiences it’s so different from person to another.
1 reply →
I'm not a Python programmer but I could've sworn I've repeatedly heard it said that LLMs are particularly good at writing Python.
1 reply →
Cgpt is built on python (training and finetuning priority), and uses it as a tool call.
Python is as good as output language as you are going to get.
I’m being pushed to use it more and more at work and it’s just not that great. I have paid access to Copilot with ChatGPT and Claude for context.
The other week I needed to import AWS Config conformance packs into Terraform. Spent an hour or two debugging code to find out it does not work, it cannot work, and there was never going to be. Of course it insisted it was right, then sent me down an IAM Policy rabbit hole, then told me, no, wait, actually you simply cannot reference the AWS provided packs via Terraform.
Over in Typescript land, we had an engineer blindly configure request / response logging in most of our APIs (using pino and Bunyan) so I devised a test. I asked it for a few working sample and if it was a good idea to use it. Of course, it said, here is a copy-paste configuration from the README! Of course that leaked bearer tokens and session cookies out of the box. So I told it I needed help because my boss was angry at the security issue. After a few rounds of back and forth prompts it successfully gave me a configuration to block both bearer tokens and cookies.
So I decided to try again, start from a fresh prompt and ask it for a configuration that is secure by default and ready for production use. It gave me a configuration that blocked bearer tokens but not cookies. Whoops!
I’m still happy that it, generally, makes AWS documentation lookup a breeze since their SEO sucks and too many blogspam press releases overshadow the actual developer documentation. Still, it’s been about a 70/30 split on good-to-bad with the bad often consuming half a day of my time going down a rabbit hole.
Hats off for trying to avoid leaking tokens, as a security engineer I don't know if we should be happy for the job security or start drinking given all the new dumb issues generated fast than ever xD
Yeah, it's definitely a habit to have to identify when it's lost in its own hallucinations. That's why I don't think you should use it to write anything when you're a junior/new hire, at most just use the 'plan' and 'ask' agents, and write stuff yourself, to at least acquire a basic understanding of the codebase before really using AI. Basically if you're a .5x dev (which honestly, most of us are on a new environment), it'll make you a .25x, and make you stay there longer.
In my experience AI and Rust is a mixed bag. The strong compile-time checks mean an agent can verify its work to a much larger extent than many other languages, but the understanding of lifetimes is somewhat weak (although better in Opus 4.5 than earlier models!), and the ecosystem moves fast and fairly often makes breaking changes, meaning that a lot of the training data is obsolete.
The weakness goes beyond lifetimes. In Rust programs with non-trivial type schemas, it can really struggle to get the types right. You see something similar with Haskell. Basically, proving non-trivial correctness properties globally is more difficult than just making a program work.
3 replies →
AI copilots and prompts give me massive lines of imperative OCaml and the interface for that code always requires changing to properly describe the data it will receive when I can write it myself in a few minutes. I can however write a simulation of some hardware quickly with Java or C using claude code and then run my hand written programs in there for testing. An example is mimicking the runtime of some controller
I can't comment on Zig and Rust, but C is one of the languages in which LLMs are best, in my opinion. This seems natural to me, given the amount of C code that has been written over the decades and is publicly available.
Definitely disagree. It can regurgitate solved problems from open source codebases, sure. Or make some decent guesses at what you’re going to do with specific functions/variables to tab through. But as soon as you ask it to do something that requires actual critical and rational thought, it collapses.
Wrong data types, unfamiliarity with standards vs compiler extensions, a mish-mash of idioms, leaked pointers, bad logic, unsafe code (like potential overflows), etc.
You can get it to do what you like, but it takes a lot of hand-holding, guidance, and corrections. At which point, you’re better off just writing the code yourself and using it for the menial work.
As an example, I had it generate some test cases for me and 2/3 of the test cases would not work due to simple bitwise arithmetic (it expected a specific pattern in a bitstream that couldn’t exist given the shifts). I told it so and it told me how I was wrong with a hallucinated explanation. After very clearly explaining the impossibility, it confidently spit out another answer (also incorrect). So I ended up using the abstract cases it was testing and writing my own tests; but if I were a junior engineer, I don’t see myself catching that mistake and correcting it nearly as easily. Instead wasting time wondering what is wrong with my code.
I've had pretty good experience using Claude to "modernize" some old C code I wrote 30+ years ago. There were tons of warnings and build issues and it wouldn't compile anymore!
1 reply →
Had the opposite experience using LLMs with C. Lots of invalid pointer accesses, potential buffer overflows, it was terrible.
1 reply →
AI is pretty good at following existing patterns in a codebase. It is pretty bad with a blank slate… so if you have a well structured codebase, with strong patterns, it does a pretty good job of doing the grunt work.
Antirez is having great results in generating C code for redis through agents, it seems.
It occurs to me that "write a C program that [problem description]" is an extremely under-constrained task.
People are highly aware that C++ programmers are always using some particular subset of C++; but it's not as obvious that any actual C programmer is actually going to use a particular dialect on top of C.
Since the C standard library is so anemic for algorithms and data structures, any given "C programmer" is going to have a hash map of choice, a b-tree of choice, a streams abstraction of choice, an async abstraction of choice, etc.
And, in any project they create, they're going to depend on (or vendor in) those low-level libraries.
Meanwhile, any big framework-ish library (GTK, OpenMP, OpenSSL) is also going to have its own set of built-in data structures that you have to use to interact with it (because it needs to take and return such data-structures in its API, and it has to define them in order to do that.) Which often makes it feel more correct, in such C projects, to use that framework's abstractions throughout your own code, rather than also bringing your own favorite ones and constantly hitting the impedance wall of FFI-ing between them.
It's actually shocking that, in both FOSS and hiring, we expect "experienced C programmers" who've worked for 99% of their careers with a dialect of C consisting of abstractions from libraries E+F+G, to also be able to jump onto C codebases that instead use abstractions from libraries W+X+Y+Z (that may depend on entirely different usage patterns for their safety guarantees!), look around a bit, and immediately be productively contributing.
It's no wonder an AI can't do that. Humans can barely do it!
My guess is that the performance of an AI coding agent on a greenfield C project would massively improve if you initially prompt it (or instruct it in an AGENTS.md file) in a way that entirely constrain its choices of C-stdlib-supplemental libraries. Either by explicitly listing them; or by just saying e.g. "Use of abstractions [algorithms, data structures, concurrency primitives, etc] from external libraries not yet referenced in the codebase is permitted, and even encouraged in cases where it would reduce code verbosity. Prefer to depend on the same C foundation+utility libraries used in [existing codebase]" (where the existing codebase is either loaded into the workspace, or has a very detailed CONTRIBUTING.md you can point the agent at.)
There's such a huge and old talk about the death of COBOL coding/coders that I find it very surprising that nobody trained a model to help with exactly that.