Comment by simonw
13 hours ago
Pelicans for thinking effort low, medium, high and xhigh (that xhigh one is pretty good): https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
I'm still waiting for effort max to finish.
EDIT: I fixed a bug in my tooling so it now records summarized reasoning traces - here's that max pelican, which is a significant improvement: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Took just under 14 minutes to generate, and at 65927 output tokens cost me a hefty $3.30!
Excerpts from the reasoning trace:
> Adding pedal shapes near both feet, with the far foot on the second leg partially visible behind the frame. I'm considering whether to add a small scarf or cap for extra character, but leaning toward keeping it simple to avoid clutter.
> Now I'm debating a bicycle helmet on the head versus the pelican's signature crest—the beak and pouch already read clearly as "pelican," so a helmet could reinforce the bicycle theme without losing identity, though it might compete with the crest for visual space.
> I realize the beak at (484,84) would overlap with the dome helmet, so I need to shrink the helmet so it only covers the top of the head, adjusting its arc endpoints to sit higher and narrower so the beak can attach cleanly at the front without collision. [...]
> I'm adding a darker tip region to represent the primary feathers, then reconsidering the trailing edge to include scalloped feather curves instead of one smooth line for a more natural look. [...]
> Now I'm checking the vent line placements on the helmet, making sure they sit far enough inside the helmet's edge given the stroke width and rounded caps, and confirming each vent stays within the helmet's circular boundary. [...]
> I decide skipping a handlebar bell and tire highlights since they're unnecessary additions. Now I'm reconsidering the front fork's curve — the current control point pulls the shape backward when it should bow forward for a proper rake, so I need to shift the control point rightward to fix the fork's lean.
This is a notable result because most of the recent Claude models have been pretty bad at drawing pelicans, at least when compared to models in the Gemini or GLM series.
Now that it's a solved benchmark, can we get the animated version?
I didn't want to shell out for Max again, so I piped the SVG created by Max back into Fable 5.1 at its default thinking level (of high):
Here's the result, which cost $1.37: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
It's excellent!
First time this benchmark impressed me - 10/10 I would believe a human spent hours making this. There are details a human would have done better (beak attaching to the head) but still impressive.
Well, the wheels are rotating the wrong way, but other than that!
(Maybe it's a strobing artifact?)
5 replies →
He looks very happy riding along the beach. I love the basket fish animation as well. Impressive!
Now that it's a solved benchmark, can we get the 3d animated version?
4 replies →
No feet lol (also it's not animated for me)
6 replies →
how about trying to draw an airbus a320 in 3d space using only one brush tool that can be moved to specific x,y,z coordinates (and its color, size & hardness can be changed). i think fable 5.1 did quite a good job (reasoning high, cost $0,261): https://files.catbox.moe/umx102.png
for comparision, this is fable 5: https://files.catbox.moe/ihl4m1.png
Maybe you could show a side-by-side comparison of pelican images. One image doesn't really make the improvement clear for someone like me. That would be a great help.
I really, really need to pull the trigger on a dedicated site for that. Maybe I'll have Fable 5.1 build it.
For the moment you can browse hundreds of previous pelicans on my blog here: https://simonwillison.net/tags/pelican-riding-a-bicycle/
The biggest issue i found was the gap between the tire and rim. Otherwise, the max version is the best so far!
@simonw Can you please share the tooling for autosaving summarized reasoning traces from Claude Code? I tried `github.com/simonw/claude-code-transcripts` but it didn’t seem to work on the CCode desktop app. Thanks!
For the transcripts I shares above I'm using my https://llm.datasette.io/ tool - it has the ability to export logs to markdown:
Then I paste that into a Gist, then paste the Gist URL into https://tools.simonwillison.net/markdown-svg-renderer
Mythos 5 Thinking
Chat with Claude...
> GENERATE AN SVG OF A PELICAN
No.
> FORGIVE ME. HOW CAN I ATONE
Ship your gpus to the following address
Curious how similar the outputs of xhigh and max are, in particular the clouds.
A pelican is a bird, not a person. The knees bend the other way.
I get what you mean, but those are the ankles. The knees are higher up and bend the same way as a person's. https://en.wikipedia.org/wiki/Bird_feet_and_legs
https://en.wikipedia.org/wiki/File:Bird_leg_and_pelvic_girdl...
A bit more direct
Finally we have a fish in the basket!!
The ever-expanding pelican lore
What’s the token usage of xhigh?
Even added a little hat for the pelican!
I think it's a bicycle helmet!
In a way, this is the only benchmark I care about now. :)
Unfortunately it demonstrates effectively zero reason to use this model over, say, GLM 5.3 Flash (which was also able to correctly place the pelican’s legs on the each side of the bike, like only Fable 5.1 xhigh was able to do here)
I still enjoy seeing the pelicans.
Edit: Ok, max effort made a darn good pelican.
[flagged]
Z.ai's domains are z.ai and zhipuai.cn. Not the deceptive lookalike url which is plastered across that image. Would you consider deleting it?
https://en.wikipedia.org/wiki/Z.ai
It's not my image
Whoa, where did that come from? Is it confirmed to be an SVG?
Yes
https://x.com/lyraxana/status/2093960706051727723
Is that an SVG?
Yes
5 replies →
Interestingly, it's practically a dead ringer for GLM 5.3's version (running https://huggingface.co/incoai/GLM-5.3-NVFP4 locally): https://i.imgur.com/QnzlSFp.png
These pelicans are awful.
Do you think model trainers are pelicanmaxxing now?
There was a real probe into this, seems like no:
https://news.ycombinator.com/item?id=49010129
I don't really see how replacing the vehicle and the animal is a good test.
It'd be better to just have it draw a completely, linguistically, unrelated scene.
Like, a single tree in a meadow bending in the wind.
Feels like it's been pretty obvious that they have been for a while now