Comment by wj

Exactly! I called that the Operations Equation and wrote a tiny bit more here about it: https://zero2data.substack.com/p/the-operations-equation

Once you are inserting models into non-creative (and software development is creative work) it is imperative to start creating your own small evals. Not just to check how different models perform in that equation, but to catch model drift over time.