Comment by simonw
3 days ago
Pelicans for low, medium, high, and xhigh: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
I had to run them via OpenRouter - I don't have access to a machine that can handle this size of model.
For comparison:
Muse Wonder: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Muse 1.3: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
And this one from Meta AI's previous open weight model, Muse Glimmer, which DID run on my laptop: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
A question I have about the pelican benchmark : why are the pelicans always cycling to the right?
The gears etc on bikes are customarily on the right, so almost all photos of bikes (eg for sale, by hobbyists) face to the right to show off that, and so it makes sense for LLMs to have such a preference.
They’re all derivatives, not originals.
because (0, 0) is on the left
In light of recent pieces like "The Emperor Pelican Has No Clothes" (https://arcfu.com/post/the-emperor-pelican-has-no-clothes/), I'd be more curious to see how 'PelicanBench' becomes more robust in the time to come -- I feel like all models are now benchmax'd on the standard pelican SVG rendering task and we need something more multi-dimensional. It would be great to have a script that prompts a model to render the pelicans in several different situations (like the linked article) that showcase some kind of broad generalizability or other sort of 'transfer learning' going on with these models that would really then be impressive and worth noting on - otherwise PelicanBench is losing its relevance.
I think it is safe to say that Muse models are not pelican-benchmaxxed.
They also are not very good lol.
It's interesting the high reasoning pelican is animated. I don't think I've seen another model do that before.
And why should it? The prompt never asks for animation.
Muse is amusing, I'll give it that.
I wouldn't touch a zuck product if my life depended on it.
Simon's posted a number of animated ones in the past. Some models really like to animate.
Did you order one of the new studios?
I really would love a browser plugin that blocks having to read about fucking pelicans on every model release.
You could unironically vibe code this with a prompt shorter than your complaint.
So make one. Not everything is for you.