← Back to context

Comment by latexr

3 months ago

> Writing both the client- and server-side for a PDF annotation editor would have taken 60 hours, maybe more.

How do you know? Seems to me you’re making the exact same estimation mistake of the people in the study.

> Instead, a combination Copilot, DeepSeek, Claude, and Gemini yielded a working prototype in under 6 hours

Six hours for a prototype using four LLMs? That is not impressive, it sounds insane and a tremendous mess that will take so long to dig out of the prototype stage it’ll effectively require a rewrite.

And why are you comparing an LLM prototype to a finished product “by hand” (I surely hope you’re not suggesting such a prototype would take sixty hours)? That is disingenuous and skewing the numbers.

> How do you know? Seems to me you’re making the exact same estimation mistake of the people in the study.

I have over 20 years of web development experience and 40 years of general experience writing software. I wrote the authors and they confirmed my thoughts:

"I totally believe it! Per the paper abstract, we find many factors driving results - and one of the factors is how experienced the developers are on the codebase, and how big/complex the codebases are.

"Given that this was a new and unfamiliar domain and new codebase, I would expect there to be much more speedup than the domain we studied!"

> Six hours for a prototype using four LLMs?

They have limits on the number of queries, so I used four different LLMs in tandem to circumvent query limits. I didn't write it four times using four different LLMs.

> it sounds insane and a tremendous mess

I posted the code. It's well organized, has few (if any) encapsulation violations, sticks to OOP quite well, works, and---if I knew the PDF.js API---would be fairly easy to maintain.

Yes, I stand by my claim that writing this annotation editor (PHP, HTML, CSS, and JS) would take me about 60 hours by hand and about 6 hours using the LLMs.