Comment by ykl
2 hours ago
I like to think of it as:
Imagine every bit of human knowledge as a discrete point within some large high dimensional space of knowledge. You can draw a big convex hull around every single point of human knowledge in a space. A LLM, being trained within this convex hull, can interpolate between any set of existing discrete points in this hull to arrive at a point which is new, but still inside of the hull. Then there are points completely outside of the hull; whether or not LLMs can reach these is IMO up for debate.
Reaching new points inside of the hull is still really useful! Many new discoveries and proofs are these new points inside of the hull; arguable _most_ useful new discoveries and proofs are these. They're things that we may not have found before, but you can arrive at by using what we already have as starting points. Many math proofs and Nobel Prize winning discoveries are these types of points. Many haven't been found yet simply because nobody has put the time or effort towards finding them; LLMs can potentially speed this up a lot.
Then there are the points completely outside of hull, which cannot be reached by extrapolation/interpolation from existing points and require genuine novel leaps. I think some candidate examples for these types of points are like, making the leap from Newtonian physics to general relativity. Demis Hassabis had a whole point about training an AI with a physics knowledge cutoff date before 1915, then showing it the orbit of Mercury and seeing if it can independently arrive at general relativity as an evaluation of whether or not something is AGI. I have my doubts that existing LLMs can make this type of leap. It’s also true that most _humans_ can’t make these leaps either; we call Einstein a genius because he alone made the leap to general relativity. But at least while most humans can’t make this type of leap, we have existence proofs that every once in a while one can; this remains to be seen with AI.
A lot of the space outside of the convex hull is just untried things. You can brute-force trying random things and checking the result and eventually learn something new. With a better heuristic, you can make better guesses and learn new things much more efficiently. There’s no reason to believe that kind of guess-and-check is outside of the reach of LLMs, or that most of our new discoveries are not found the same way.
I come back to something like this idea when I consider the distinction being made that LLMs can only combine and interpolate between points in their training material. I could write a brute-force program that just used an English dictionary to produce every possible one-billion-gazillion word permutation of the words within, with no respect for rules of language, and chances are there would be some provable, testable, novel insight somewhere in the results if you had the time to sift through and validate all of it. LLMs seem like a tool that can search that space more effectively than any we've had before.
> There’s no reason to believe that kind of guess-and-check is outside of the reach of LLMs
This doesn't make any sense, by their nature they can't "guess-and-check" things outside their training set.
I like this construction, but I don’t think you take it far enough.
If you have a multi dimensional space, and you are trying to compute which points lie “inside” some boundary, there are large areas that will be bounded by some dimensions but not others. This is interesting because it means if you have a section bounded by dimensions A, B, and C but not D, you could still place a point in D, and doing so then changes your overall bounds.
I think this is how much of human knowledge has progressed (maybe all non-observational knowledge). We make observations that create points, and then we derive points within the created space, and that changes the derivable space, and we derive more points.
I don’t see why AI could do the same (other than technical limitations related to learning and memory).