Comment by kikdij

Interestingly, it _is_ possible to get the exact same output from the exact same input with LLMs. For example, with llama.cpp/llama-server, you can set the temperature to 0, define a seed for the randomizer, turn off caching so it doesn't mess with the inference, and you would get the exact same result every time. But it's largely an academic distinction, because 1/ change a letter in your prompt, and it's over… and you're certainly going to want to change your prompt at some point, and 2/ model publishers have recommendation about the temperature setting, usually between 0.7 and 1, so your reproducible results… might be terrible at solving actual problems.

That being say, I really like the idea of committing our prompts. I can see a possible future in which that would be our work: write precise specs so that you can generate the program again with an other model, in an other programming language, and get the same behavior. Which sounds a hell lot like… a RFC. :)

We may be the culprit here, by the way. Maybe the only thing missing is us taking "prompt engineering" more seriously. It would be specification writing, really, or protocol design. I think it might already be possible, from an experience I had. I made Codex write a POSIX vi implementation from POSIX ex/vi specs. It took it a week-end, but I now have a POSIX vi, with only added on top multilevel undo/redo and syntax coloration, and without the baggage and instability of nvi2, I'm an happy man. I've been using it for more than a month and still have to find a fault to its POSIX implementation (and there are _a lot_ of details in POSIX vi specs). I think models are already there, when it comes to implemented behavior predictability, it's just us who are not precise enough in our specifications.