From a comment by j2kun https://news.ycombinator.com/item?id=49000833 , someone asked Fable and there was an almost counterexample in 2d but it uses division too. [Instead of f=x^2+7xy they have something like f=x^2+7x/y so it's not a polynomial.] As far as I know, nobody know what trick to make to avoid that division. It looks like the new trick was to use a third variable to avoid the division. Note that the implementation of the trick is not straightforward. The almost counterexample was sitting around for almost 30 years, and nobody knew how to fix it.
From another old comment, someone else was trying to find a counterexample with 16 variables using a computer to make thousands of attempts and failed. So it's far from obvious that the trick to add a variable solves the problems.
Close to impossible. This is a famous enough problem that anyone who understands what they are doing generally would pretty immediately recognise the significance of the counterexample if shown it.
The best part is that we can't know the answer to that.
The necessary precursors to the counter example where definitively in the training set, otherwise the LLM wouldn't know how math works, but at the same time, we can't tell whether there were mathematicians who got 90% of the way, then gave up and the LLM just did the last 10%.
Extremely high. Or at least several partial solutions that can be smooshed together.
LLMs really do still just reassemble things in their training data. There’s just a lot of it now, people anthropomorphise and struggle visualising large things. Some people say it’s truly reasoning but hit a topic that is under represented in the data of any LLM and it’ll transport you very quickly back a couple of years and ruin the illusion quickly.
The problem being that I don't think there's a definite proof that any of human thinking is more than a sum of high-granularity partial solutions that can be put together.
It could be that with enough tokens, big enough context window, and ability to dig out the relevant partials, many such thought processes could be simulated.
From a comment by j2kun https://news.ycombinator.com/item?id=49000833 , someone asked Fable and there was an almost counterexample in 2d but it uses division too. [Instead of f=x^2+7xy they have something like f=x^2+7x/y so it's not a polynomial.] As far as I know, nobody know what trick to make to avoid that division. It looks like the new trick was to use a third variable to avoid the division. Note that the implementation of the trick is not straightforward. The almost counterexample was sitting around for almost 30 years, and nobody knew how to fix it.
From another old comment, someone else was trying to find a counterexample with 16 variables using a computer to make thousands of attempts and failed. So it's far from obvious that the trick to add a variable solves the problems.
Close to impossible. This is a famous enough problem that anyone who understands what they are doing generally would pretty immediately recognise the significance of the counterexample if shown it.
I'd wager close to 0%
The best part is that we can't know the answer to that.
The necessary precursors to the counter example where definitively in the training set, otherwise the LLM wouldn't know how math works, but at the same time, we can't tell whether there were mathematicians who got 90% of the way, then gave up and the LLM just did the last 10%.
Anthropic could audit the model to find the answer. It’s telling that they won’t do this.
Extremely high. Or at least several partial solutions that can be smooshed together.
LLMs really do still just reassemble things in their training data. There’s just a lot of it now, people anthropomorphise and struggle visualising large things. Some people say it’s truly reasoning but hit a topic that is under represented in the data of any LLM and it’ll transport you very quickly back a couple of years and ruin the illusion quickly.
The problem being that I don't think there's a definite proof that any of human thinking is more than a sum of high-granularity partial solutions that can be put together.
It could be that with enough tokens, big enough context window, and ability to dig out the relevant partials, many such thought processes could be simulated.
[dead]
I think you’re just speculating.