Comment by krupan

Thanks for writing that. It reminds me that there are many things we build and they work (for some definition of work) even though we don't fully understand them.

Did the first people that made fire understand it? You mentioned bridge building. How many bridges have failed for unknown (at the time) reasons? Heck, are we sure that every feature we put into a bridge design is necessary or why it's necessary? Repeat this thought for everything humans have created. Large software projects are difficult to reason about. You'll often find code that works because of a delightfully surprising combination of misunderstandings. When humans try to modify a complex system to solve one problem they almost always introduce new behavior, the law of unintended consequences.

All that being said, we usually don't get anywhere without at least a basic understanding of why doing X leads to Y. The first humans that made fire had probably observed the way fires started before they set out to make their own. Same with bridges and cars and computers.

So yes, you are absolutely correct that nobody fully understands how AI/LLMs work. But also, we kinda do understand. But also also, we're probably at a stage where we are building bridges that are going to collapse, boilers that will explode, or computer programs that are one unanticipated input away from seg faulting.