Comment by tptacek
10 months ago
I have a lot of sympathy for the author's position but I may have missed the point in the article where he explained why clarity of writing and genuineness of human expression was so vital to a robotics class. It's one thing for an instructor to appreciate those things; another for them to confound their own didactic purpose with them. This point seems obvious enough that I feel like I must have missed something.
As always, I reject wholeheartedly what this skeptical article has to say about LLMs and programming. It takes the (common) perspective of "vibe coders", people who literally don't care what code says as long as something that runs comes out the other side. But smart, professional programmers use LLMs in different ways; in particular, they review and demand alterations to the output, the same way you would doing code review on a team.
I think they summed it up well in the section "Why do we write, anyway?" — they nowhere claimed it was vital for students' success in a robotic class. On the contrary as they title a subsection there with "If it’s worth doing, it’s worth doing *badly*" (emphasis mine) — so what they are looking for is to peer into the author's mind and their original thoughts.
The implication there is that this is acceptable to pass a robotics class, and potentially this gives them more information about students' comprehension to further improve their instruction and teaching ("...that they have some kind of internal understanding to share").
On that second point, I have yet to see someone demonstrate a "smart, professional programmer use LLMs" in a way where it produces high quality output in their area of expertise, while improving their efficiency and thus saving time for them (compared to them just using a good, old IDE)!
A couple of examples from influential open source developers: Adam Wathan (Tailwind) agreeing with Mitchell Hashimoto that LLMs are making them more productive. "in their area of expertise" is not obvious from this post alone, but I am pretty confident from the way they talk about it that they are not exclusively using LLMs on side projects where they're inexpert.
https://x.com/adamwathan/status/1911845073286803923
Armin Ronacher also talks about using LLMs quite a bit, but I don't have as good of an example from his tweets of him straightforwardly saying "yes, they are useful to me!"
"Saying" vs "demonstrating": I am careful with the words I am using.
It's not just nitpicking: maybe it's only their perception they are more productive? Or maybe they are. But saying it is not convincing enough for me.
2 replies →
Who have you asked? What has been your experience observing professionals using LLMs?
Is "what has been my experience" not implied in what I am still waiting for — "someone using an LLM to produce high quality code in the field of their expertise in less time than without an LLM"?
So, observing a couple of my colleagues (I am an engineering manager, but have switched back and forth between management and IC roles for the last ~20 years), I've seen them either produce crap, or spend so much time tuning the prompts that it would have been faster to do it without an LLM. They mostly used Github Copilot or ChatGPT (most recent versions as of last few months ago).
I am also keeping out a keen eye for any examples of this (on HN in particular), but it usually turns out things like https://news.ycombinator.com/item?id=43573755
Again, I am not saying it's not being done, but I have struggled to find someone who would demonstrate it happen in a convincing enough fashion — I am really trying to imagine how I would best incorporate this into my daily non-work programming activities, so I'd love to see a few examples of someone using it effectively.
8 replies →