Comment by hashmap

this landed precisely on like 3 weird bugs ive been hitting and solving in different stupid ways for dealing with things like sgd collapsing too many good answers into one bad answer, and gave me a real direction to try to fix the link missing in my own ml stuff. what timing. i have tried analytic solutions too and they're useful for like mapping prompts into memory geometry but from there ive ended up still having to use sgd. cause i think what happens is, sgd teaches the neural net both the geometry and how to navigate it. if you just teleport to the answer it doesnt learn how to walk.