Comment by cscheid
4 days ago
> All non-trivial properties of neural networks are undecidable.
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
Not just that, but we can solve quintics in the almost[1] general case to arbitrary precision, providing we are willing to accept approximate solutions.
[1] I exclude things that are truly perverse such as quintics with coefficients that won't fit on any physically realizable computer.
Back in college I played a similar game by drawing septagons with a straightedge and compass. They're not constructable, but a length within a three significant figures is easily constructed, and that's more accurate than I am with a compass.
That's good to know. It's very useful information.