Comment by theturtletalks

for a lot of things ChatGPT has 100% plateaued, especially if you are just using it as a "better Google" search engine. ChatGPT still makes mistakes, makes things up occasionally, etc. that's the whole jagged frontier thing. what recent LLMs have truly achieved breakthrough in are fields that have results that are relatively cheap to verify, such as maths, coding, etc. so people outside of these fields will have disconnects with where LLMs really are right now.

also very important to note that the massive increase in capability in these fields doesn't necessarily immediately translate to increase in productivity. because the code and math proofs these LLMs write are legit _alien_ to us. if you just want code that do something (e.g. a port of Bloodborne that runs on PC), then they can do it. but if you want something human can understand and maintain, that's going to be a nightmare. and that'll surely slow down adoption.

A terminal also tells the model when it failed. Exit code, stderr, a stack trace. it reads that and tries again.

The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK. - Slack puts {"ok": false} inside a 200. - Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.

The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.