Comment by fallingknife
5 months ago
This is correct. Most people hear about AI from two sources, AI companies and journalists. Both have an incentive to make it sound more powerful than it is.
On the other hand this thing got 83% on a test I got 47% on...
> On the other hand this thing got 83% on a test I got 47% on
Easy to do when it can memorize the answers in its training data and didn't get drunk while reviewing the textbook (that last part might just be me).
The Olympiad questions are puzzles, so you can't memorise the answers. To do well you need to both remember the foundations and exercise reasoning. They are written to be slightly novel to test this and not the same every year.
This thing also hallucinated a test directly into a function when I asked it to use a different data structure, which is not something I ever recall doing during all my years of tests and schooling.
Must have been quite the hangover to prevent your recalling this.
On the other other hand, it had the perfect recall of the collective knowledge of mankind at its metaphorical fingertips.
Is that supposed to make it sound more powerful or less powerful than it really is?