Comment by simonw
11 hours ago
llm -m meta-ai/muse-spark-1.3 "Generate an SVG of a pelican riding a bicycle"
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
4.2266 cents, 38 seconds.
For comparison here's Muse Spark 1.2, which animated it without me asking it to: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
The 1.3 one is definitely better - better bicycle frame, better wing, better pelican hat.
UPDATE: Here's another one with five pelicans for each of the five Muse Spark 1.3 reasoning levels: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
The most expensive was reasoning level xhigh - 7.5 cents, 1m34s.
And I ran five pelicans at all reasoning levels for 1.2 as well, here: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
I was interviewed for a job as a software developer last week and they asked me to draw a picture of a pelican riding a bicycle.
Aced it, got the job as a senior software engineer.
The interviewers afterwards said "it is SO refreshing to find a software developer who actually knows how to code - never seen such a high performance focused, well built pelican on a bike - you have the skills we need".
I interviewed as a software developer at LinkedIn. The interviewer asked me to demonstrate my prompting skills, so I had AI write an article about what the recent death of my father taught me about B2B SaaS. Reading it brought tears to his eyes so he hired me on the spot.
I interviewed as a software developer at Meta. They asked me to do a add legs to the player in a VR world. I couldn't do it. They hired me anyway.
1 reply →
You should really spam that link here to show your dad’s memory lives on.
Sorry for your loss.
1 reply →
"software developer"...you keep using that word. I do not think it means what you think it means.
Is this for real
2 replies →
I also aced my interview by focussing on pelicancode problems, instead of leetcode problems.
There are still job interviews?
If you could actually hand write SVG code on the spot that looked like a realistic pelican riding a bike I would want to hire you for SOMETHING.
Or placed in an asylum next to the people who designed XML
We were hand writing PostScript code that drew pelicans at job interviews in the 90s, then they sent it to a printer a stored the page in a file drawer /s
"I was interviewed for a job as a software developer last week and they asked me to draw a picture of a pelican riding a bicycle. Aced it, got the job as a senior software engineer."
That is the best joke I have heard this year. Ready for a stand-up comedy special. Or a song. Superb!
Little did you know "they" were secretly harvesting data so their models can draw the best pelicans because someone keeps benchmarking them.
Obviously you failed a trick question. Pelicans can’t ride bikes.
You should post your source code you wrote here… ;)
Is there a reason these pelicans always have roughly the same composition (side-view, 2d, biking right, flat ground beneath, etc)? I don't see any of that detailed in the prompt, yet they all seem to generate roughly the same image of differing quality.
The more generic your prompt, the more generic the response. It's a regression to the "mean" of the training data aka GIGO for AI.
It's like when you ask your average person off the street to draw a house - it'll almost always be square with a triangle roof, one door, and two windows.
In the pelican/bike example, it's probably a bit of a self-perpetuating snowball too. If the earliest examples were bike left-to-right, flat ground, etc. then they are also being scraped up in future LLMs.
Well, all the LLMs are being trained on previous pelicans, so they look the same.
1 reply →
as a kid I did them like this. nobody told me to do that. are we all so similar?
8 replies →
Not when rendered via POV-Ray:
https://blog.nawaz.org/posts/2025/Oct/pelican-on-a-bike-rayt...
I plan to update it with more pelicans from all the models released since.
(Spoiler alert: They haven't improved much since then).
Ohh, horizontal wheels. They’re about as good as I expected, models have pretty bad spatial awareness. I would expect Fable to be a bit better than old models, though.
Wow, I actually had this exact idea. I was specifically curious as to how well a given LLM could understand a DSL that hasn't changed much in a couple decades and doesn't have nearly as many examples to learn from online. Seems like it did alright, all things considered.
I wonder how a multi-modal model would do with a harness and tool calling? Specifically a "render" command that produced an image output enabling it to iterate. (Well I see you did this manually with gemini 2.5 pro but I still think it would be interesting to explore various harness setups.)
> GPT-5.1 Codex
> monstrosity
What are you talking about? That's clearly a sci-fi pelican on a hoverboard (successor of the humble bicycle) wearing a visor. Truly visionary.
It's really interesting, isn't it? They almost always cycle from left to right - but I have had a few which cycle in the other direction.
The 2D / flat ground feels reasonable for a SVG, which implies a vector illustration.
It's my impression that it's common in western culture, where text is read left to right, and timelines are visualized as going from left to right, to also animate things going from left to right, since westerners thus have an instinct that "right = forward", so it "feels right" (familiar). I wonder to which degree this is reflected in the training data? And if you'd be more likely to get left-facing pelicans if you prompted it in Hebrew, Arabic or another right-to-left language?
Someone studied this (among other thigns): https://dylancastillo.co/posts/pelicanmaxxing.html . Pelicans on bikes always face right in this test, but other animals on other transportation methods sometimes face left.
I was going to ask the exact same question earlier but deleted it after thinking “I’m sure Simon has done some sort of discussion on this.” Since it does seem novel to you, too, it would be really interesting to read more about this phenomenon.
Search Google Images for "bicycle". Almost all bicycle product shots are staged the same way: side view, going left-to-right. It makes sense to me that given that skew in the training data, the model grounds itself in the bicycle.
and furthermore, this is because the drivetrain is ~always on the right side of the bike - if you want to inspect or admire a bicycle you look at the right side, as you might look under the hood of a car.
(Why the drivetrain is on the right, I don't know. But most bike parts follow open standards so it's quite entrenched.)
4 replies →
Product shots yes, people riding them its more like 50/50. Also if you search for a specific bicycle race you'll find more going right to left.
The canonical view of a bicycle is facing right. Usually, people want to draw/photograph/depict the side of the bicycle with the running gear, which is on the right side of the frame for historical reasons.
The thing that distinguishes pelicans from other birds does so most strongly in profile. If you're looking straight at one, the throat pouch would be hidden by the beak.
I bet if it instead had something to do with black widow spiders we'd find that we're most often looking at the bottom of the spider's abdomen, regardless of whatever non-spider-like activity is supplied.
Yes. It's because you are asking it to generate an image of a pelican riding a bicycle. If someone asked you to draw a pelican riding a bycycle, would you interpret that to mean using 3d photorealism? LLMs follow conventions. The convention for an animal riding a bike is to create a childish 2d line drawing.
well it is svg, it is doing it from circles and lines as primitives, it wants to do it simply and kind of builds the whole thing hierarchically. Making it 3d is way more complicated (as the POV example shows) and the prompt doesn't say 3d anyway
I’m a firm believer in pelicanmaxxing.
They’re all so close in proportions.
Sun is missing a few rays and not wearing sunglasses.
Yes, I do a thing where I ask the machine to generate responses in the form of a lizard talking to a cat. The lizard is always a green gecko and the cat is always orange, which I never specify.
Is there a reason these pelicans always have roughly the same composition
Because they're computers. They don't have an imagination and the ability to create things from whole cloth the way humans do.
Much like a mother pelican, they regurgitate what they've been fed.
I'm waiting for the models to start responding with "Oh hi Simon!"
Simon, at this point I really wonder if teams aren’t gaming this. You should pick a random animal doing a random thing every time.
They're still not yet at the point where pelicanmaxxing is the best way to win this benchmark. Earlier models sucked because their SVG skills sucked. Newer models are likely better because more/better SVG models are being added to their training data.
We should just consider the pelican bench as saturated and mostly meaningless.
But the general improvements are obvious. Get them to draw something very different (e.g. a wifi rotary phone with a peeled banana handset and a coiled cable, or a pink tennis ball with strawberry seeds and a reset button) and you can see that improvements are not narrowly tailored.
I'm not sure why it had to have the pelican wearing a red scarf seeing as that was not in the prompt
If you have a grading rubric, huge points off for adding arms instead of using the wings as arms!
I think it's hilarious that this detail is enough for me to dismiss looking into the model, but here we are, and it is.
Did any LLM so far draw pelican knees correctly and have them bend in opposite direction from human knees? Knees of many animals bend opposite to humans.
Did any LLM draw the front bicycle wheel correctly? ie. center of front wheel slightly AHEAD of steering wheel axis. This is done for bicycle stability.
https://en.wikipedia.org/wiki/Bird_feet_and_legs#/media/File...
Bird knees bend same way human ones do
It's clear they mean the 'exposed' joint where humans assume the knees, and where one can see the leg bend. Technically you're correct, but it's just that. Please answer in better faith instead of well akshually.
is there a reason there are so many common base decorative elements across pelicans on bicycles? For instance, there's a lot hats/helmets and scarfs/capes across models.
FYI, your renderer breaks with error "git api access error 403", rate limiting error from git, when using cloudflare vpn.
I am guessing its not super common, but it happens just so you know.
Is there any point anymore regarding this svg test? I would not be surprised if in the training they're fine tuned for this task too
It would be very embarrassing for any lab to benchmaxx the pelican on bicycle svg prompt, since it would be very easy to detect it by varying the prompt.
The amount of discussion around it means that the test and all the reviews of results, images, approaches etc are implicitly included in training data.
It’s not deliberate “benchmaxxing” but things that are discussed a lot online are naturally things that LLMs learn better.
1 reply →
You win this thread's prize:
https://news.ycombinator.com/item?id=49538333
It also works as extremely effective engagement farming, for lack of a better phrase
Has any ab tried to game this yet and just made the most amazing pelican by hand and always reply with that?
I see no point having these pelicans used for anything related model qualification.
At this point the only thing they're useful for is visualizing the differences between effort levels and roughly tracking the progression of models within a specific model family. And they still do that really well!
All of the links show "Error: Gist API returned 403".
next, try: "generate an svg of a human hand". this is a prompt where many models fail imo.
excellent thread
What does the mean pelican look like at this point?
Also 3X token use vs. 1.2
Red eyes and a tattoo?
Absolutely BRUTAL! :)
Thank you for doing this, I love your benchmark the most!
For all the comments of "I'm sure they're fine-tuning for pelicans": https://dylancastillo.co/posts/pelicanmaxxing.html
lol
Definitely an upgrade over 1.2
[flagged]
Someone did care enough to create a throwaway account to vent here, it seems.
how tf you can have green names with negative karma ???
I wonder, given Simons reputation in AI benchmarking, whether model providers try to train or tweak their models to perform better at drawing bicycles and pelicans?