Comment by gizajob
18 hours ago
“ There were tasks I could have done in 20 minutes easily, that took 5 minutes of an AI agent, and then 2 days for me to review.”
This is using AI for productivity in any domain, in a nutshell. I just wrote a book using Claude as an experiment, and while the thing got done and it was an amazing tool and a great experience, what I’m left with is a book where every line needs rewriting, there are logical inconsistencies throughout, and the style is so bad it should actually just be binned rather than rewritten.
> I just wrote a book using Claude as an experiment, and while the thing got done and it was an amazing tool and a great experience, what I’m left with is a book where every line needs rewriting, there are logical inconsistencies throughout, and the style is so bad it should actually just be binned rather than rewritten.
How was this a great experience if what was produced needed such extensive changes that your own assessment is that it should be thrown out? At what point is the necessary rework so much that the thing being reworked didn't really contribute much to the end product at all?
It was really interesting in having an assistant who was completely on the ball and up to speed, never forgot, could pick up from where we left off days ago, could produce and summarise arguments… but the issue was that the amazing assistant could only really work one chapter at a time even with a plan, and changing one chapter would often break things said in other chapters, and their style of writing was such that the resorted to cliches, bad metaphors, and generally bad or repetitive prose. So the grammar was fine, the style was bad to the point where the whole work would not be something I could have my name on or near. I really could come back to it, just having to retype everything in markdown made me see that the project had legs and the topic is sufficient for a book length treatment, but even with the help of Claude I have a few months work to do to get it done and out there. Just it’s a different few months work to the work I would have had without Claude. I could also have attempted to pay Claude thousands of dollars to see if it worked better, but the book is not likely to recoup what that would cost.
That's not surprising, I've used LLMs to write several chapters of a book and it's many times more difficult than getting them to write working code. It's not a task that they're well optimized for but the biggest hindrance is that there are practically no tools that allow you to validate prose especially at the lengths required for a book.
Getting a good writing style out of them requires careful prompting and many corrections, their default writing style(s) are so highly reinforced by training that they will always tend to drift back to them. Maintaining continuity requires you to create a lot of documentation outside of the text itself. It's a much more manual process than working on a codebase where you've set up a lot of automation and tooling that allows them to check their own work.
Edit: it's also worth noting that many LLMs have gotten much worse at writing prose as they have gotten better at writing code.
This is the way. Basically, what determines when to use AI or not is what you enjoy about the creative process. Do you only care about the result, use AI, and then spend hours (or days) fixing errors, tweaking, rewriting, using other AI:s to validate and improve the first AI etc. You get the result, and your activity consist of arguing with AI:s.
If you enjoy the craft and the creative process of actually coming up with new thoughts, instead of relying on the probabilistic combinations of thoughts of others, then you can just as well do it yourself and have full control of the process.
I use AI for low value work with dead lines, where the customers don't really care about the result either. For the golden services and customer engagements were I can tell the customer cares deeply, I use little to no AI, and then get a deep sense of fulfillment due to a job well done.
Sadly, a great deal of applied business programming in modern capitalism doesn't give you the option of enjoying the creative process.
In effect, everything is classified as: "low value work with dead lines, where the customers don't really care about the result either."
So you use AI when it's all right to produce slop. That actually seems reasonable.
I don’t see how that’s possible. When I am doing a comprehensive code review with rigorous functional testing of another developer’s work, it usually takes me around half as long to review as the developer took to write it. That includes the back and forth of MR issues and fixes. If AI writes it why would it take 100x longer to review than write it? At worst case it’s just a draft of something I can develop myself, so it shouldn’t take longer than 20 minutes. At best case it is a code review exercise so it takes me half as long.
AI can produce for you an amazing jpeg of an oil painting you’ve ideated together. You could print it out on a large format printer but you’re still only holding a printout of an AI oil painting. To actually make a piece of art that’s worth being on a wall and standing as an artwork, then you still have to paint the painting using oil paint.
Maybe robot arms can solve this last part one day.
I’m not an artist I’m a software developer, so the tools I use don’t matter, the outcomes do. If I am completing my tickets and the code is high quality and well designed then I am doing my job.
Depends on how the AI is used. If used intelligently, for short, readable code snippets, reviewing and understand is easy.
If used naively, where AI spits out 1000s of lines of code, that is far from perfect, written in a style that might not be what you are used to, it can take longer to parse, than if your colleague of 2 years wrote it.
I noticed lately that recent LLMs write very short sentences, sprinkling so many periods over one paragraph. I'm pretty sure that this is some new regression that we're having with new models.
Have you noticed it, too?
Hmmm I was using Claude Opus for the most part. I didn’t notice any particular shortness in the length of the sentences, but that’s also an issue in itself – the whole thing is just average fine-ness: the sentences are fine, the paragraphs are fine, everything is generic and inoffensive and fine. But at the same time it was giving huge amounts of pushback when I was getting it to interrogate logical faults of certain domains, to the point where I was having to argue with and even convince an AI using data and even its own analysis that there was cause to talk about certain things from a certain angle (can appreciate I’m being vague about which things). It can’t write complex, deeply-claused sentences (or won’t). And it also wants to write snappy little concluding sentences that make a fairly well-argued piece of sociological analysis read like Sex and the City or something.
And just like that, the book got abandoned.
N.b. some of its analysis and laying out of faults in arguments was actually pellucid and brilliant, it can’t be denied. Just it comes with prose that can’t really be used for anything. And even on another occasion when I got it to help me redraft and extend a different book of mine, then it randomly and consistently started stripping out all the stylistic flourishes out of my sentences, to the point where it couldn’t notice that word choices were very deliberate and actually set up little punchlines and logical payoffs paragraphs or chapters hence. And even when I explained and showed it what it was doing, it was like “ahh that’s so clever and brilliant” but just continued to do the same thing.
That's a side effect of people bashing the em-dash and the labs rushing to correct. The period is the most common replacement (along with some minor tweaks to sentence structure), so now that models are being RL'd away from the em-dash the models are overusing a new construction.
That's stupid. People didn't bash the em dash because it was an em dash. They bashed it because it made it obvious it was AI. If your product is so good why are your customers ashamed of using it?
1 reply →
What made the editing process intractable?
It’s not that it’s intractable, and I genuinely enjoyed the process working with Claude. But responding to the initial post – using Claude I managed to do three months work in about two weeks, but at the end of that I then realised I probably had three months of improvements and rewrites and corrections left to go if it was going to be to be a book worth anyone else reading.