Comment by fph
20 days ago
The authors mention that before publications they tested these questions on Gemini and GPT, so they have been available to the two biggest players already; they have a head start.
20 days ago
The authors mention that before publications they tested these questions on Gemini and GPT, so they have been available to the two biggest players already; they have a head start.
Looks like very sloppy research.
I don't think it's that serious...it's an interesting experiment that assumes people will take it in good faith. The idea is also of course to attach the transcript log and how you prompted the LLM so that anyone can attempt to reproduce if they wish.
If you want to do this rigorously, you should run it as a competition like the guys at the AI-MO Prize are doing on Kaggle.
That way you get all the necessary data.
I still think this is bro science.
2 replies →