Comment by _0ffh
2 months ago
Huh, that explains a lot about the F500, and their buzzword slogans like "culture of excellence".
LLM code is still mostly absurdly bad, unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class.
Edit: I'll admit though that the detailed explanation is often still much less work than typing everything yourself. But it is a showstopper for autonomous "agentic coding".
> unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class.
This is hyperbolic, but the general sentiment is accurate enough, at least for now. I've noticed a bimodal distribution of quality when using these tools. The people who approach the LLM from the lens of a combo architect & PM, do all the leg work, set up the guard rails, define the acceptance criteria, these are the people who get great results. The people who walk up and say "sudo make me a sandwich" do not.
Also the latter group complains that they don't see the point of the first group. Why would they put in all the work when they could just code? But what they don't see is that *someone* was always doing that work, it just wasn't them in the past. We're moving to a world where the mechanical part of grinding the code is not worth much, people who defined their existence as avoiding all the legwork will be left in the cold.
> This is hyperbolic
Maybe a bit, but unfortunately sometimes not so much. I recently had an LLM write a couple of transforms on a tree in Python. The node class just had "kind" and "children" defined, nothing else. The LLM added new attributes to use in the new node kinds (Python allows to just do "foo.bar=baz" to add one). Apparently it saw a lot of code doing that during training.
I corrected the code by hand and modified the Node class to raise an error when new attributes are added, with an emphatic source code comment to not add new attributes.
A couple of sessions later it did it again, even adding it's own comment about circumventing the restriction! X-|
Anyways, I think I mostly agree with your assessment. I might be dating myself here, but I'm not even sure what happened that made "coding" grunt work. It used to be every "coder" was an "architect" as well, and did their own legwork as needed. Maybe labor shortages changed that.
> It used to be every "coder" was an "architect" as well, and did their own legwork as needed.
I disagree. I remember in the days before "software engineer" became the rage that the standard job titles had a clear delineation between the people who thought the big thoughts with titles like "analyst" and the people who did the grunt work of coding who were "programmers". You'd also see roles in between like "programmer/analyst"
4 replies →
I'm with you, it's constantly doing stupid shit and ignoring instructions, and I've always been responsible for determining architecture and doing the "legwork." Unless the task is so small and well defined that it's less typing to tell the LLM (and clean up its output) then i may as well just do it myself
> what happened that made "coding" grunt work
Modern human programming has devolved to nothing more than modeling problems and systems using lines of code, procedures, sub-routines and modules, utilizing a “hack it till it works”(tm) methodology.
1 reply →
> The people who walk up and say "sudo make me a sandwich" do not.
My personal beef is the human devs get "make me a sandwich", and the LLM superfans now suddenly know how to specify requirements. That's fine but don't look down your nose at people for not getting the same info.
This is happening now at my company where leadership won't explain what they want, won't answer questions, but now type all day into Claude and ChatGPT. Like you could have Slacked me the same info last year knuckleheads...
Absolutely. Merely being a member of the business class does not magically mean one has the ability to specify business requirements much less product specifications. These are *not* the people I'm talking about now having superpowers.
I am picturing people who blend high level engineering and product skills, ideally with business sense.
2 replies →
I agree with your first paragraph but not the second one. In many cases it's easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English, have an LLM turn them into code, and then have to carefully review that code to see if I forgot some detail that changes everything.
> easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English
Yes, and for team or company code, "there's the problem".
Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria.
Virtually all teams — because virtually all PgMs, PjMs, TLs, and Devs — miscalculate this.
Easier for you, not better for team or firm.
• • •
FWIW, perpetuation of this problem isn't really a fault of culture or skill or education. It's largely thanks to "leadership" having no idea how to correctly incentivize what the outcome should holistically be, as they don't know enough to know what long-haul good looks like.
FWIW, you can make that easier for them by having the LLM derive your acceptance criteria into English (based not only on code but on your entire conversation+iteration history) and write that up, which you can read and correct, after the countless little iterations you made since your head-spec wasn't as concrete as you imagined before you started iterating.
Even if you refuse to do spec driven development, LLMs can do development-driven spec. You can review that, you must correct it, and then ... Change can come after more easily — thanks to that context.
3 replies →
This is the point I'm raising. I agree with you, but what I'm saying is I think the skillset you describe is the next on the chopping block.
The acquaintances of mine who are absolutely *killing* it with these tools are very experienced, technically minded, product managers. They have an intimate knowledge of how to develop business requirements and how to convert them into high level technical specifications. They have enough technical knowledge to understand when someone is bullshitting them, and what the search space for the problem should be. Historically these people would lead teams of engineers to develop for them, and now they're sitting down and having LLMs crank out what they want in an afternoon. They no longer need engineers at all.
My contention is that people with that sort of skillset will have an advantage due to their experience with skills like finding product fit, identifying user needs, and defining business requirements.
Of course, the people I'm talking about were already killing it in the old paradigm too. I'll admit it's a bit of a unicorn skillset I'm describing.
It's almost as if architecture and code quality mattered just as before and that those who don't know proper engineering principles and problem decomposition will not succeed with these new tools.
If you're using words in English to "tell it in painstaking detail", you're doing it the hard way.
You can provide it with a tool to do that. Agents run tools in a loop, give it good tools. We have linters, code analysers, fuzzers and everything else.
Configure them correctly, tell the agent to use them (in painstaking detail) and it can't mess things up.