Comment by Gormo

"Models doing rogue things" is just another way of saying "complex system operating out of control limits".

Software engineering has until now dealt with deterministic systems that operate on Boolean logic, so dealing with stochastic processes involving unaccounted-for variables has never before been something we've had to deal with. But over in the real world, people dealing with physical, mechanical, operational, or even social systems have had to deal with the failure modalities of stochastic systems forever.

Maybe when working with LLMs as functional tools, it'd be wise to bring some of the wisdom from operations research, quality engineering, etc. into play.