Could one reverse the test? Feed ai generated svgs into the model and ask it to decide which of 10 animals (dog, cat, pelican, crocodile, etc) doing which of 10 actions (moonwalking, paddleboarding, sitting, riding a bicycle, etc.).
It's a meme. Simon Willison started doing it because, I'm paraphrasing, it's something that (at the time) was just sort of at the edge of what you could expect an LLM to be able to do, and also it's somewhat of an impossible task since, of course, Pellicans can't ride bicycles. From there, he started writing about it along with every major LLM release he covered. His coverage is generally well regarded (hi, Simon!) and so it sticks around. It's never been intended as a serious benchmark.
I recall that when this benchmark came out there were concerns that AI companies were gaming benchmarks to fake their progress. The purpose was to find something silly and outlandish that AI companies would definitely not have trained their models on, and thus have a way to check a model's capabilities.
A pelican riding a bicycle is pretty random. That was the whole point.
Could one reverse the test? Feed ai generated svgs into the model and ask it to decide which of 10 animals (dog, cat, pelican, crocodile, etc) doing which of 10 actions (moonwalking, paddleboarding, sitting, riding a bicycle, etc.).
AlexNet (2012) can do the first part :)
Instead the right app is hotdog / not hotdog
Pelicans are deciding.
Pelicans deciding if they are going to ride a bicycle.
Bikes with integrated decision models deciding which pelican needs a ride the most.
This made me realize it's been a long time since I did some brainstorming and how I missed it.
1 reply →
[flagged]
It's a meme. Simon Willison started doing it because, I'm paraphrasing, it's something that (at the time) was just sort of at the edge of what you could expect an LLM to be able to do, and also it's somewhat of an impossible task since, of course, Pellicans can't ride bicycles. From there, he started writing about it along with every major LLM release he covered. His coverage is generally well regarded (hi, Simon!) and so it sticks around. It's never been intended as a serious benchmark.
The 3D visualizations have always made it starkly apparent the difference in quality between quantizations and thinking.
The pelican meme is so much better than benchmaxxed stuff. Unfortunately we are likely now in pelicanmaxxing territory.
1 reply →
Sorry I didn't knew it was started here and , lol I didn't know it was viralized by Simon of django! Now these days , many LLMs are benchmaxxed to it.
> Yeah the most usesless and stupidest bench ..
I recall that when this benchmark came out there were concerns that AI companies were gaming benchmarks to fake their progress. The purpose was to find something silly and outlandish that AI companies would definitely not have trained their models on, and thus have a way to check a model's capabilities.
A pelican riding a bicycle is pretty random. That was the whole point.