Comment by simonw
2 days ago
The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!
2 days ago
The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!
Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model.
In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.
Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.
You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass.
Blue sky and green grass aren't that surprising, but the color and direction are interesting.
When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like which direction, color of bicycle, general pelican geometry etc. It will be interesting to see if other creatures end up with coincidentally similar design choices or if that's unique to the pelican-bicycle combination.
In photography (and probably art in general), there's a composition "rule" to frame moving subjects from left to right.
So the direction may not be that interesting!
15 replies →
What's interesting is that given the fairly general and short in length prompt for the test, none of the models are attempting things like more discrete details of the bike. Such as showing V-brakes or dual 160mm disc rotors, rear derailleur, water bottle in a bottle cage, panniers, lights, saddlebag, the rider wearing a helmet, or other details that might be found on as vague a description as "a bicycle".
It'd be hard to fully compare, but I think a truly random "creature-vehicle" along side the pelican test would catch who's gaming and who's not.
I'd also enjoy the absurdism of "Herring on a pogostick"
1 reply →
There's a bias in the direction all things face. You can ask these models to generate a thing animal, car etc and you will notice that 90% of them will converge towards the same sort of results. If you ask for something rotating, 90% of them will rotate right and a few odd ones will rotate left.
I have done some variation of the other animals, also for something more tricky where they need to calculate things, I ask them to draw an SVG at a certain angle.
For example: "generate an SVG of a chessboard seen from a 45 degree angle slightly higher POV" or "generate an SVG of a basketball court from a TV broadcast perspective".
I find Gemini is still the best at creating SVGs.
The art styling is more or less uniform too.
I haven't seen many AI works that produces a pelican on a bicycle done in a "Ligne Claire" style, for example.
I guess AI's narrows down the output probability space drastically and converge on some agreed upon aesthetics. Works great for computer programs but bad for art.
I thought my joke post was silly and then I read new comments and I'm like, "I didn't try hard enough" lol
Bicycle color, grass color and sky color are all part of the prompt.
>Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground
2 replies →
> the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model
You would not expect that to happen if the models trained on the unrecognizable mess, right?
> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts
And the labs clearly did focus on improving image rendering.
> they have a uniform style
SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.
I’m not suggesting Simon’s pelicans in the dataset are having a meaningful impact. I’m expecting that a company like ScaleAI has a product along the lines of “benchmax dataset: SimonW’s Pelican on Bikes test” which is a private curated series of well-drawn SVGs of animals riding vehicles for training and RL.
2 replies →
> Moreover, they have a uniform style, even though your prompt doesn’t ask for one.
This shouldn’t really come as a surprise, particularly to anyone who’s used diffusion models. The same thing happens when you ask an LLM for a short story [1] without providing any specific details.
Even cranking up the temperature or top_p values is no panacea. The more generic your prompt, the more pedestrian the response.
[1] - https://news.ycombinator.com/item?id=42093394
If you’ve been keeping track of all of the pelicans, there is actually stylistic differences - sometimes pretty big differences as far as watercolors go. It’s an SVG so I’m not sure what you’re looking for there. Most look the same because the prompt is to make a pelican on a bicycle as an SVG. It’s not some giant image prompt.
> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts
That doesn't seem right. I use these models as research assistants when writing lots of random blog posts (including in economically ~useless areas like the history of contra dance) and Fable 5 is a serious improvement (when I don't get downgraded!) over Opus 4.6-4.8 which was a serious improvement over Opus 4.
Watercolors in SVG?
Also, I'd assume the ideal output for an underspecified, generic prompt is the most expected, generic result. Not something that defaults off the rails with creative license.
being able to draw a picture of a pelican is really cool and it requires intelligence but i don't think it's a good measure of improving capabilities of these models nor AGI. we don't have to spend so much breath on it.
It's just a gut feeling, but I think you're running a (very slow) distributed hill-climbing algorithm. LLM1 generates an SVG. You post it online, with commentary on what is good/bad about it. LLM2 consumes the SVG alongside your commentary, and produces a slightly different SVG. Rinse, repeat.
I'm saying an example of what not to do is still an example.
Simon - has no one told you about the Willison-Pelican Scaling Law?
```
if is_willison_pelican_blog_post:
[redacted]
```
You haven't seen their final form [1]
[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.
And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]
[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.
PS: have you physically seen a pelican in real life? (not a joke)
> PS: have you physically seen a pelican in real life? (not a joke)
We have several thousand living 15 minutes walk from our house. I recently started adding my wildlife photography (from iNaturalist) to my blog, so I'm posting several new pelican photos a week at the moment: https://simonwillison.net/search/?q=pelican&type=beat%3Asigh...
Simon - thank you for not dismissing it (and surviving the text that came before the question).
I asked because I genuinely feel that the % of people working on some of the most important technology these days - things such as these 'strangely shaped tools' (to borrow from nearcyan) - large language models - the younger generation (folks in their early/mid 20s) - it is not unlikely that they have not physically seen the meatspace version of whatever digital correspondence of it that is being packed into latent space.
After all, why waste time going to the SF or Oakland zoo? One can just check Simon's latest pelican blog post and skip the zoo trip - the harnesses are waiting.
Yes, I see your point.
Your pelican output is thus both in the training set and yet still outside the capability of the model architecture.
And so you are tracking both the capability of the training and also the capability of the querying!
When you receive your first outstanding pelican it will track a gain of capability.
(btw I first mentioned simonw-pelican-into-training-set in May 2025 on twitter.)
My 3D-egyptology-explainer showed a massive uplift for Kimi K3 and this tracks a much improved 3D capability.
I agree with that. I think, in particular, all the broken bike frames associated with "pelican on a bike" probably make it harder for LLMs to render correct bike frames.
...even with the glut of pelicans, aren't there still far more images of actual bikes (with correct frames) available to train on?
Perhaps I'm underestimating the number of pelicans(?!)
At this point I am simply interested in how much longer you're gonna ride this schtick
I'm a deep believer in commitment to the bit. https://simonwillison.net/tags/pelican-riding-a-bicycle/
as long as he gets paid for that
What does good look like?
The dedicated text-to-image models all produce good illustrations of pelicans riding bicycles. Here's one I got from OpenAI's gpt-image-2 just the other day: https://simonwillison.net/2026/Jul/14/pedalican/
Have you tried asking for an SVG with the same level of detail you specified in that prompt, though? https://github.com/simonw/pedalican/blob/main/run/prompts/ro...
I'd be interested to see what comes out, but it also highlights an curious prompt-control-comparison question
Try prompts that convert the gif to svg? Maybe the models should do that internally… start with their image model first and then make an svg.