Comment by theletterf
16 days ago
Author here: If the LLM revolution helps us get more accessible and better docs, I, for one, welcome it.
Edit: I guess some commenters misunderstood my message. I'm saying that by serving also the needs of LLMs we might get more resources to improve docs overall.
We've seen this story many times before. Remember UML and Rational Rose, waterfall, big up front design, low/no code services etc. In every case the premise is "pay a few 'big brains' to write the requirements in some 'higher level language' and tools/low skill and low pay engineers can automatically generate the code."
It turns out, to describe a system in enough detail to implement the system, you need to use a programming language. For the system to be performant and secure you need well educated high skill engineers. LLMs aren't going to change that.
Anyway, this is tacitly declaring LLM bankruptcy - the LLM can't understand what to do by reading the most precise specification, the code, so we're going to provide less specific instructions and it will do better?
> It turns out, to describe a system in enough detail to implement the system, you need to use a programming language.
Go back to design patterns. Not the Gang of Four, rather the book where the name and concept was lifted from.
What you will find is that implementations are impacted by factors that are not always intuitive without ancillary information.
It's clear when there is a cowpath through a campus, and the need for a sidewalk becomes apparent. It's not so clear when that happens in code because it often isnt linear. It's why documentation is essential.
"Agile" has made this worse, because the why is often lost, meetings or offline chats lead to tickets with the what and not the why. It's great that those breadcrumbs are linked through commits but the devil is in the details. Even when all the connections exist you often have to chase them through layers of systems and pry them out of people ... emails, old slack messages, paper notes, a photo of a white board.
Oh, I remember. I'm an OpenAPI fan, for example. This time it feels a bit different, though. It won't erase the requirement of having high skill engineers, nor the need to have tech writers create docs, but it might indeed help focus on docs under a different light: not as a post-factum, often neglected artifact, but as a design document. I'm talking of actual user docs here, not PRFAQs or specs.
It always feels a bit different doesn't it. This time it's though. Maybe. No definitely. Will see. We're waiting for AGI anyway, right?
1 reply →
The difference is that LLMs are more capable than the average coder, and rapidly improving. If you give a seasoned developer a loose spec, (s)he'll deal with the ambiguity.
Right, because LLMs cannot learn from their prior work in a project nor can they read a large code base for every prompt. So it can be helpful to provide it documents for understanding the project and also the code base layouts immediately in its context, but you can also generate a lot of that and just fix it up.
the existence of "legacy code" i think is sufficient evidence that the code is not a sufficiently detailed description of a system. if it were, legacy code would not exist, because any engineer could simply read the code and maintain it like the original authors, without needing to say, talk to stake holders or prior documentation, etc.
we're still going to need well educated, skilled engineers, but the LLM context can store both lower fidelity details - eg. "this needs a list, but who cares if its an array or a linked list" and detail not covered by the running code "this looks like it could be a set, but we didnt do that because ..."
hell, semantic web even. we wouldn't need AI
> more accessible and better docs
Easy now. You might be skilled in documentation, but most developers write docs like they write code. For the most part all you are only going to get the program written twice, once in natural language and once in a programming language. In which case, you could have simply read the code in the first place (or had an LLM explain the code in natural language, if you can't read code for some reason).
> you are going to get the program written twice, once in natural language and once in a programming language.
How is this a bad thing? Personally, I'm not superhuman and more readily understand natural language.
If I have the choice between documentation explaining what a function does and reading the code, I'm going to just read the docs every time. Unless I have reason to think something is screwy or having an intricate understanding is critical to my job.
If you get paid by the hour then go for it, but I don't have time to go line by line through library code.
> How is this a bad thing?
It is not good or bad. I don't understand your question.
> If I have the choice between documentation explaining what a function does and reading the code, I'm going to just read the docs every time.
The apparent "choice", given the context under which the discussion is taking place, is writing your program in natural language and have an LLM transcribe it to a programming language, or writing your program in a programming language and having an LLM transcribe it to natural language.
1 reply →
It does not give us better docs. The claude.md files are a performative action for the employer. No one will write APUE level documentation because everything is rewritten constantly in those code bases anyway and no one either has the skills or is rewarded to write like that any more.
That's why I feel tech writers need step in and help.