Comment by simonw
1 day ago
I generated pelicans riding bicycles on both thinking level low and thinking level high:
https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304...
The high one is notably better - the bicycle frame is the correct shape, unlike thinking level low.
For comparison, here's Opus 4.7: https://gist.github.com/simonw/afcb19addf3f38eb1996e1ebe749c...
It's pretty safe to say that AI will be used on the battlefield making real life and death decisions before it will be able to render a decent pelican on a bike in SVG.
It already has been and this has been widely written about. AI was used to identify and prioritize targets for the US to bomb in Iran.
Here's an article from 2 months ago for example: https://www.theguardian.com/technology/commentisfree/2026/ma...
It was also implicated in the bombing of a girls elementary school which left 168 dead. The US did a "triple tap" to kill any first responders.
https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam...
https://www.theguardian.com/technology/2026/apr/01/dont-blam...
I read the article and it doesn’t say it was used for targeting or prioritizing?
> Neither Claude nor any other LLMs detects targets, processes radar, fuses sensor data or pairs weapons to targets. LLMs are late additions to Palantir’s ecosystem. In late 2024, years after the core system was operational, Palantir added an LLM layer – this is where Claude sits – that lets analysts search and summarise intelligence reports in plain English
There’s a lot of humans in that loop who make those decisions.
6 replies →
I think it's beyond decent. I don't understand how people are not more impressed by this. Just a few years ago the only expectation would be garbled nonsense.
Haha, yeah. I tried for it to create a SVG with scissors and it was hopelessly overwhelmed. I think at least the SVG design niche will be safe a little while longer
the battlefield sounds much easier. worst case scenario you kill somebody, but that's what you're trying to do anyways.
if you kill somebody while trying to render a pelican on a bicycle it's a real problem.
In many battlefield scenarios, there is more than one "somebody" on it. The "somebody" that you kill might not be the "somebody" that you intended to kill.
Depending on the how pelicans are created, it is entirely possible to indirectly kill "somebody" due to the externalised costs of global warming etc.
"shift left" on the battlefield. break down those silos. if you have to ask for permission it's already too late. remember the goal. find the bottlenecks in your system and remove them.
I think that's a fair tradeoff. There's no way I'm going back to writing code by hand again. No one deserves that.
Heh? How long were you writing "code by hand" before?
3 replies →
> the bicycle frame is the correct shape
No, the handlebar is wrong. The handle bar is rotating the frame instead of rotating the front wheel. The handle bar should be mounted on the same line as the front wheel is.
Hopefully 4.9 will read my comments :)
Could be an extremely high angle stem that just happens to match the downtube angle.
Maybe the pelican is just riding a road bike/gravel bike
I bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so:
https://www.gianlucagimini.it/portfolio-item/velocipedia/
Turns out even humans can be pretty bad at drawing bicycles :)
As an aside, some of the renders have only a single side connection to the wheel and that is a valid bike design, the Cannondale Lefty front fork only has a left leg:
https://duckduckgo.com/?q=cannondale+lefty&iar=images&t=ffab
On a new model release, you can guarantee two things are in the replies to Simon. One is your link, the other is "surely the models are being trained on this now"
Sure, but no one is trying to force art from most people into about every area in the economy where anyone ever pays for something visual. If you asked professional artists to draw a realistic bicycle, I'm guessing few of them would try to just randomly guess what the mechanical parts looked like
> The most unintelligible drawing has also the most unintelligible handwriting. It was made by a doctor.
Haha
But if you need to draw a bicycle, you wouldn’t pick a random person in the street. You would hire an artist and you’d be guaranteed to have at least a believable one if not a perfect rendering.
No guarantees is why LLM is akin to gambling. Every new context is essentially picking someone out of the crowd.
Sadly I think the correlation between this benchmark and performance is starting to break down imo. Still a legendary idea will be remembered and ingrained in the models forever haha
Glad to see that the "high thinking" level adds a helmet. Always a smart choice.
And yet some people doubt Anthropic's commitment to AI safety
Here's pelicans in all of the thinking levels - low, medium, high, xhigh, max
https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Max seems to me to be notably better than the others.
low: yolo
medium: redesign bike so peli can reach bars
high: redesign bike so peli can rest on frame
xhigh: yolo
max: big peli reach bars
I like the way the max pelican has a stern look on his face
Is the output on the max level meant to be missing?
I just fixed that (force refresh). It hit my default 8,000 output token limit, it worked when I bumped that up.
For max I used 25 input, 17,167 output which cost me 43 cents! https://www.llm-prices.com/#it=25&ot=17167&ic=5&oc=25&sel=cl...
You've peed in the pool Simon, this has to be a part of the internal evals by now! You got to try something new - maybe a panda in a canoe?
If these were in the internal evals then the output would be much better. The 4.8 pelicans are pretty meh
Click the link
I really like that thinking level high gave the pelican a helmet.
Hey simonw I love your test, do you think using thinking level "max" makes sense for this test? I would love to see the results about it.
I don't think the API supports "max" as an option, that might just be a Claude Code harness thing.
UPDATE: My mistake, the API does support max. I added a max one at the bottom of this page (cost 43 cents): https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
The legend.
It's funny that we've reached the level where LLMs draw more correct bikes than any random person
Simon, is your pelican test really captures differences among models or should you at least try like 10 times or something to average the random effects
I've been meaning to do a "run 3 times and pick the best" version for quite a while, I should really pull the trigger on that one. Currently it's one-shot only.
You could run 3 times and overlay/average the images to show how consistent they are
Best-of-3 would be cheating, ruin the test, middle of 3 makes more sense
3 replies →
The vast majority (if not all) of these make it impossible to turn, among other fun things. Only out of curiosity, have you tried prompting further with how a bike must operate to see if it does the right thing?
tried it myself, not much of difference
https://gist.github.com/fendy3002/3026a8c4d67d1301666ec40fc0...
looks like the model already trained well on both bicycle and pelicans
I actually like the 4.7 the most, interestingly enough. Not like you can "objectively" weight artistic output like this.
I find the most miraculous thing about 4.7 to be that the pelican is facing left, wonder why the right facing everything is so ubiquitous in these images.
This happened to me in elementary school. We were doing fingerpaintings using plasticine. After all the bikes were hung on the wall, mine was racing the other way... Somehow it really stuck with me.
What do you think it means?
It's facing left but looking right...
Profound political commentary?
[dead]
I don't see how a frame without a headtube can be "the correct shape".
thanks for always providing this very much on time. I'm wondering what the next, harder challenge could be? Maybe some animated svg?
That little red hat on hard mode is sending me. 4.8 has whimsy
Is the "opossum riding an e-scooter" benchmark in the works for Opus 4.8? ;)
Good call, it's cute: https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304... - but nothing like GLM-5.1: shttps://static.simonwillison.net/static/2026/glm-possum-esco...
[flagged]
For comparison, what's GPT-5.5 producing today?
The reasoning xhigh one is pretty solid: https://simonwillison.net/2026/Apr/23/gpt-5-5/#and-some-peli...
Lends credence to my vibe-based assertion that GPT-5.5 > Opus 4.7 (and now 4.8), which is why I've cancelled my Claude plan. Opus 4.8 is them seeing it reflected in their own numbers and having to pull stopgap measures to avoid falling behind while they embargo Mythos.
Am I allowed to say that pelican's little helmet is adorable? I can't provide a strong computational proof, or even a shred of anecdata...
...but that pelican's little helmet is adorable.
Eventually the frontier model folks are going to pick up on your pelican on a bike test and bake-in flawless results for that particular request.
4.7 reigns supreme IMO.