Comment by podlp

I tried Elixir a few months back with several different models (GPT, Claude, and Gemini). I’m not an Elixir or BEAM developer, but the results were quite poor. I rarely got it to generate syntactically correct Elixir (let alone idiomatic). It often hallucinated standard library functions that didn’t exist. Since I had very little prior experience, steering the models didn’t go well. I’ve since been using them for JS/ TS, Kotlin/ Java, and a few other tasks where I’m much more familiar.

My takeaway was that these models excel at popular languages where there’s ample training material, but struggle where the languages change rapidly or are relatively “niche.” I’m sure they’ve since gotten better, so perhaps my perception is already out of date.