Comment by jdlshore
7 hours ago
This is an amazing article. The problems it describes are exactly what we found when building a production system that used LLMs to (most of the time) produce reliable results. Extensive tests are necessary, and stakeholders have no idea how their suggestions fail in production. They just see the handful of times they tried something and had it work, not the long tail of cursed results. (“How hard can it be? Why don’t you just…”)
We didn’t get to the point of self-built prompts, as the article suggests, but it’s an intriguing idea.
This guy is on the ball with the problem. Totally correct: Like my friend Coda says all the time: "the textual nature of prompts leads us to take the intentional stance towards systems which aren’t conscious, and thus miss the essential nature of their non-meaning."
I don't know if his solution (""We should all go insane building interlocking evaluation and optimization pipelines, instead.") would be the long-term solution. Instead perhaps something could be trained into the models, i.e., he is describing a process at inference time that could be done at training time. To make their weird errors less frequent / make them more human.
cf. https://arxiv.org/abs/2008.04071 "On Controllability of AI" However, as I said, you can't make it perfect but you can make it better. (You can't make humans fully aligned with human society's interest anyway, including the humans controlling the nukes.)