> The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal before being told to continue, and every few tool call outputs add comment lines that refer to SVGs of pelicans (and, perhaps, how a møøse bit my sister once). And, at the end, once it's 800k tokens deep into context, it's asked to produce an SVG of a pelican and is evaluated against both the pelican and the completion and efficiency of the task.
You're only as good as your ability to solve problems in the midst of an SVG pelican attack.
Ask it to write a program that outputs SVGs of animals using human modes of transportation, then run the program with "pelican" and "bicycle" as inputs.
> How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.
The pelican benchmark is exactly what's wrong with hiring in technology.
It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.
It's got nothing to do with what most people actually do when they're working..
AI companies claim their products are generalists though, and that they can do a good job on anything you give them, so you can't say what people will be doing with it. "Generate an SVG of an bird on a bicycle" is a corner case certainly but if a candidate interviewing for a role claims they can handle the corner cases then it's totally fair to assess them on that.
Besides, if you move up one layer to "how good is AI at generating valid SVG markup of non-obvious things", pelican on a bike is actually a good test.
Just as even a counterpoint to this, I have asked the LLMs to attempt to generate SVG icons for websites. Even though I have requested things much simpler than a Pelican, they have all tended to do quite poorly in my examples.
Because of this, I presume the Pelican has been in the training data for at least a year+.
The models are very useful, I am afraid they have fundamental limitations though generalizing (it is just hard to evaluate effectively). So it will just be whack-a-mole "can your model do X", and there will always be a new X.
Goodhart’s law is the problem, not the metric itself. Also LLMs do not have any visual generation skills, so its idea of a pelican looks like purely linguistic, unlike diffusion models. That we get decent results at all from an LLM outputting SVG files of random things is just nuts to me.
I think they're less and less advertised as true generalists these days, as they pivot to profits that obviously lie (for the time being) first and foremost in agentic coding. It's no longer unusual to see regressions in terms of more stiff prose due to the strong tuning towards coding, or how they structure their response. And prose is a LLM's home turf! Instead, progress in agentic coding capability is usually the headline feature, the headline benchmark, etc etc. At least looking at Anthropic, Google, OpenAI. There are of course other LLM's.
So then add a dash of cybersecurity and medical use and that's basically it. No "closer to AGI" advertising. I'd say the 2026 development has in fact been the opposite; optimizing AI for niches where there is most potential for profits and that your description died in circa GPT-5 era.
In fact, this problem (for this test) is also stated by the pelican test author:
"The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
Personally I'd consider the three middle ones to be failing, in the typical "Gemini/Google" fashion in that the model is doing more than what the prompt asks. The prompt asks for SVG, yet the model is providing more.
Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.
By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!
Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model.
In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.
Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.
It's just a gut feeling, but I think you're running a (very slow) distributed hill-climbing algorithm. LLM1 generates an SVG. You post it online, with commentary on what is good/bad about it. LLM2 consumes the SVG alongside your commentary, and produces a slightly different SVG. Rinse, repeat.
I'm saying an example of what not to do is still an example.
Simon - has no one told you about the Willison-Pelican Scaling Law?
```
if is_willison_pelican_blog_post:
[redacted]
```
You haven't seen their final form [1]
[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.
And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]
[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.
PS: have you physically seen a pelican in real life? (not a joke)
I agree with that. I think, in particular, all the broken bike frames associated with "pelican on a bike" probably make it harder for LLMs to render correct bike frames.
Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.
From the article it doesn't even sound like he cares about pelicans at all, and doesn't think they are a good way to compare models anymore ... but people are used to seeing the test now, and it does serve as a common "hello world" unit of work.
It's incredible you can't reason to see if pelican on a bike is a thing. It's not! This has been discussed to death. You can ask any model to generate anything. Generate an SVG of earthworm and a robin boxing. Guess what? The smarter the model the better the image, doesn't matter if it's a vision model or not. I rolled my eyes at this eval when I first saw it, then I tried various ridiculous things and noticed a very strong correlation. Things that are absolutely not in the training set.
More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.
They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.
Respectfully, did you? The comment was specific to doubting the believe simonw has that labs are not training [0] specifically for this task, which is exactly what simonw wrote in the post [1], that it is a believe of his that they don't. He did not mention any kind of evidence or any piece of information that would indicate that the commenter didn't read the blog post.
Did you read either the post or the comment it was referencing?
On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.
This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.
More like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”
We do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to. I'll leave the contrasts between LLMs and people to the well-written sibling comments.
One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is better" might just be which run you picked for it. Would love to see 8 runs per model side by side. I bet for two close models, the gap between runs is about as big as the gap between the models.
I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition.
If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions.
Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.
The disconnection between pelican quality and overall model quality is interesting. I initially assumed that since pre-training is when a model gets its general skill that it happened around when RL started to really differentiate models. That is higher quality pre-trains result in higher quality pelicans, but RL is unlikely to touch pelican quality. However the fact that GLM 5.2 beats GPT 5.6 and Claude Fable puts a damper on that idea.
My only guess is that GLM 5.2 was specifically RLed for SVG generation and that resulted in superior performance.
Wait, the user asked for a SVG of a pelican riding a bicycle. That doesn’t make sense, and I need to think about whether this is a legitimate request.
The user is asking to to generate an innocent and mundane graphic, possibly as part of a test.
But wait, pelicans cannot ride bicycles! A pelican is a water bird, and bicycles are designed to be ridden humans. Something alarming may be happening here, could this a jailbreaking attempt?
I need to reconsider and reread the user’s request, “make me a svg of a pelican riding a bicycle”. That is a perfectly innocent and legitimate task, as well as popular “benchmark” on social media communities, so I will continue. I need to continue to be on alert and watch out for potential jailbreaking attempts.
I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.
Or they just don't actually have any compute access restrictions of significance? Chinese companies can just go use those GPUs in neighboring countries that aren't export-restricted, like Malaysia. Like ByteDance openly did: https://www.tomshardware.com/pc-components/gpus/chinas-byted...
And that's not even considering just smuggling the GPUs in by eg buying them in Singapore.
AI-specific chips also seem to be on the easier side to design & create relative to high performance CPUs & GPUs, so there's no particular reason to expect Chinese domestic designs to continuously lag behind. They have access to the same fabs, after all
Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.
Firstly, the export-restricted GB202s (e.g. 5090, RTX 6000 Pro Blackwell) are fabled in TSMC, and then packaged/made in... China before they supposedly have to be sold out (by US law; but not by Chinese law). You can immediately see the problem there.
Secondly, despite the supposed 'crackdowns' and et al, NVIDIA and their channel partners pretty much will sell to anyone in countries like Singapore without any questions.
Third, there's human "smugglers" who just physically carry em on trips, and Chinese customs is obviously not going to care about the US's laws on Chinese soil.
Training and serving large models does require increasingly more compute, though. (The Chinese labs have clearly found some massive optimizations, but my point was that you'd think at some point even those optimizations wouldn't be enough to keep up with exponentially increasing model sizes.)
The Chinese just saved the world economy by draining their absurdly enormous oil storage reserves nobody knew they had, wouldn't surprise me if they had lots of hidden compute too.
Huawei Ascend chips were used to train DeepSeek v4 over 4 months ago, and they shared their kernel with the other Chinese labs. China also has their own DDR5 fabs.
> The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic’s Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date
Interesting to see that prices are converging to an "equilibrium price" regardless of being US or Chinese.
The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.
Anyone have any idea what the architecture/vendors they are using for inference/compute?
Getting the compute to run inference for multi-trillion parameter models at any sort of scale and performance is daunting. There are a handful of vendors that have systems that can do this (~ Nvidia NVl-72 class) that pretty much only the frontier labs and hyperscalers effectively have access to.
One article... every time. And the only reason it gets any traction is because of who the author is -- not because of anything substantively useful. Do you think this whole "pelican on a bicycle" would have blown up if, say, you were the first?
If you look at https://news.ycombinator.com/from?site=simonwillison.net you'll see that I submitted just one out of the last thirty articles from my site that were submitted to Hacker News - and the one I submitted failed to gain any votes.
Do any of the vision models render the SVG and look at the result.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the image looked great and didn't need any tweaks. Maybe I should try it again, there have been some newer models since I first tried it. And yeah, maybe worth trying with bigger models. But I have found that models aren't necessarily the best at visual reasoning and review, even with a vision loop. Their lack of visual reasoning is part of why they still have trouble with things like ARC-AGI-3.
I've found much better luck giving it an audit check-list, including some steers like: are there any visual glitches or SVG bugs, are the colours consistent, etc.
That's kind-of why I don't think they're doing that. Anything beyond something that works with a simple design templates looks, well, like they tried to do too much with a simple design template.
3T is impressive, but parameter count seems to be less important than I thought.
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.
Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.
> The number of the experts tells you about the diversity of its skills.
Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.
Time to replace a pelican with a drawing of an original electronic schematic. Let it choose components, vary power requirements, input voltage, the output signal.
It will be valuable to have two types of benchmarks: ones that evolve alongside the models and ones that never change. You probably can't get historical stability and resistance to flooding and training on at least some parts of it from the same test
Wild that we still haven't figured out how to make good benchmarks. What we really need is a way to properly quantify what makes a codebases architecture good, and then evaluate architecture of generated codebases, or evaluate refactors of existing ones.
Also, a way to evaluate a models ability to remove dead code, clean up slop, reorganize, etc.
None of the existing benchmarks test any of the things that truly matter. They were relevant when models struggled to one-shot functions, but we're so beyond that point right now, yet the industry has not kept up.
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one.
Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
As mentioned elsewhere, the benchmark introduces bad pelicans in the training set. What I'm curious about however, if it's possible for a human artist to "poison" the benchmark by releasing some really good pelicans svgs and have all future models output their version.
Another day, another model and another pelican :-)
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.
Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).
Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed.
My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities (as we've already seen in iron smelting and clean energy), sanctions-busting, gamed benchmarks, outright theft -- or a combination of the above.
I think this is one of the few cases where there isn't a grift. It's open source, open research, there's not much to hide?
Honestly official statements are pretty tame, it's the people who spin them for media headline clicks that are warping reality
Invest in energy, manufacturing, and education (ie. your own people) for 75 years and people will look for a trick card up your sleeve and accuse you of cheating when your 7th of the world population has a 7th of the world's genius
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.
Sorry, how again is this the end of the frontier labs?
According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.
Well, with the actions of the US government, for every business that does not exclusively operate in the US, they have now added _supplier risk_ to US companies.
Even as a paying customer, even as an enterprise, your access to US models may be turned off at any time for arbitrary reasons, including someone mis-understanding "Please fix this [open source] code" (which contained security vulnerabilities that were fixed) as a jailbreak.
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.
> they're comparing to similar capability llm models, not humans
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
I am not a fan of this benchmark, nor the interpretation of Simon's. Can you draw a pelican riding a bike, and that would pass with flying colors if ranked by a diverse set of human judges? If not, you have your answer r.e. test credibility.
> The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal before being told to continue, and every few tool call outputs add comment lines that refer to SVGs of pelicans (and, perhaps, how a møøse bit my sister once). And, at the end, once it's 800k tokens deep into context, it's asked to produce an SVG of a pelican and is evaluated against both the pelican and the completion and efficiency of the task.
You're only as good as your ability to solve problems in the midst of an SVG pelican attack.
Ask it to write a program that outputs SVGs of animals using human modes of transportation, then run the program with "pelican" and "bicycle" as inputs.
this is probably about $5 bucks in codex . worth introspecting why nobody seems excited to run it
> How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.
This is quite possibly reasoning-effort prompt which is injected before the opening <think> token whenever you set a custom reasoning effort, see e.g. DeepSeek-V4 max mode prompt: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
The pelican benchmark is exactly what's wrong with hiring in technology.
It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.
Exactly what someone without nine years of 10X pelican drawing experience would say
It's got nothing to do with what most people actually do when they're working..
AI companies claim their products are generalists though, and that they can do a good job on anything you give them, so you can't say what people will be doing with it. "Generate an SVG of an bird on a bicycle" is a corner case certainly but if a candidate interviewing for a role claims they can handle the corner cases then it's totally fair to assess them on that.
Besides, if you move up one layer to "how good is AI at generating valid SVG markup of non-obvious things", pelican on a bike is actually a good test.
Just as even a counterpoint to this, I have asked the LLMs to attempt to generate SVG icons for websites. Even though I have requested things much simpler than a Pelican, they have all tended to do quite poorly in my examples.
Because of this, I presume the Pelican has been in the training data for at least a year+.
The models are very useful, I am afraid they have fundamental limitations though generalizing (it is just hard to evaluate effectively). So it will just be whack-a-mole "can your model do X", and there will always be a new X.
1 reply →
Goodhart’s law is the problem, not the metric itself. Also LLMs do not have any visual generation skills, so its idea of a pelican looks like purely linguistic, unlike diffusion models. That we get decent results at all from an LLM outputting SVG files of random things is just nuts to me.
I think they're less and less advertised as true generalists these days, as they pivot to profits that obviously lie (for the time being) first and foremost in agentic coding. It's no longer unusual to see regressions in terms of more stiff prose due to the strong tuning towards coding, or how they structure their response. And prose is a LLM's home turf! Instead, progress in agentic coding capability is usually the headline feature, the headline benchmark, etc etc. At least looking at Anthropic, Google, OpenAI. There are of course other LLM's.
So then add a dash of cybersecurity and medical use and that's basically it. No "closer to AGI" advertising. I'd say the 2026 development has in fact been the opposite; optimizing AI for niches where there is most potential for profits and that your description died in circa GPT-5 era.
In fact, this problem (for this test) is also stated by the pelican test author:
"The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
So don’t go using pelicans to compare models!"
2 replies →
It is a simple prompt that packs a lot:
* operates an absurd prompt
* involves SVG coding knowledge, generates a source code artifact
* involves world knowledge (what is a pelican? What is a bicycle? What does each do?” How are each constructed?”)
* when rendered, the coding artifact expresses an image that makes sense to us perceptually, including color and spatial relationships
* different models and settings have different output so it can be used as an evaluation scheme
That said I wouldn’t choose a model based on this! Just like some brain teaser shouldn’t determine employment eligibility.
Hi, experienced pelican here. No.
Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
Kimi is cheapest by 5x but also slowest by 2x
https://9gpyw4uxr2.evvl.io/
Personally I'd consider the three middle ones to be failing, in the typical "Gemini/Google" fashion in that the model is doing more than what the prompt asks. The prompt asks for SVG, yet the model is providing more.
Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.
By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!
Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model.
In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.
Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.
37 replies →
It's just a gut feeling, but I think you're running a (very slow) distributed hill-climbing algorithm. LLM1 generates an SVG. You post it online, with commentary on what is good/bad about it. LLM2 consumes the SVG alongside your commentary, and produces a slightly different SVG. Rinse, repeat.
I'm saying an example of what not to do is still an example.
Simon - has no one told you about the Willison-Pelican Scaling Law?
```
if is_willison_pelican_blog_post:
[redacted]
```
You haven't seen their final form [1]
[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.
And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]
[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.
PS: have you physically seen a pelican in real life? (not a joke)
2 replies →
Yes, I see your point.
Your pelican output is thus both in the training set and yet still outside the capability of the model architecture.
And so you are tracking both the capability of the training and also the capability of the querying!
When you receive your first outstanding pelican it will track a gain of capability.
(btw I first mentioned simonw-pelican-into-training-set in May 2025 on twitter.)
My 3D-egyptology-explainer showed a massive uplift for Kimi K3 and this tracks a much improved 3D capability.
I agree with that. I think, in particular, all the broken bike frames associated with "pelican on a bike" probably make it harder for LLMs to render correct bike frames.
1 reply →
At this point I am simply interested in how much longer you're gonna ride this schtick
2 replies →
What does good look like?
3 replies →
Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.
From the article it doesn't even sound like he cares about pelicans at all, and doesn't think they are a good way to compare models anymore ... but people are used to seeing the test now, and it does serve as a common "hello world" unit of work.
Pelicans and bikes can be in the training set without them training for this specific benchmark.
Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?
7 replies →
It's incredible you can't reason to see if pelican on a bike is a thing. It's not! This has been discussed to death. You can ask any model to generate anything. Generate an SVG of earthworm and a robin boxing. Guess what? The smarter the model the better the image, doesn't matter if it's a vision model or not. I rolled my eyes at this eval when I first saw it, then I tried various ridiculous things and noticed a very strong correlation. Things that are absolutely not in the training set.
More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.
This reminded me about the news cycle last year that we were running out of training data (and how silly that was)
Yeah I asked Nano Banana to make a render of our company office and was scarily accurate
They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.
Did you read the post? It's not even that long. He explicitly mentions this...
Are they responding to: “I’m still not convinced that labs are training for the benchmark—if they were, I’d expect much better results.”
1 reply →
Clearly not. There's a subset of HN users who rush to post this same thing every single time.
5 replies →
Respectfully, did you? The comment was specific to doubting the believe simonw has that labs are not training [0] specifically for this task, which is exactly what simonw wrote in the post [1], that it is a believe of his that they don't. He did not mention any kind of evidence or any piece of information that would indicate that the commenter didn't read the blog post.
Did you read either the post or the comment it was referencing?
On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.
[0] ... incredible Simon still believes ...
[1] I’m still not convinced that labs ....
4 replies →
It's incredible people still discuss the pelicans... But then again, the ad just works.
A person from Google famously put on her linkedin that her job was to optimize SVG for Gemini 3.0.
SVG output is useful, though. I often ask whatever LLM I have open to generate placeholder icons whenever I need them.
[dead]
Imagine if we applied this train of logic to humans.
"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.
More like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”
We do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to. I'll leave the contrasts between LLMs and people to the well-written sibling comments.
Except, of course, LLMs are not humans, and they do not learn or "reason" in a way which even remotely resembles humans.
Plus obviously humans can still overfit to a specific style of test.
One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is better" might just be which run you picked for it. Would love to see 8 runs per model side by side. I bet for two close models, the gap between runs is about as big as the gap between the models.
I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition.
I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-...
I probably should spend some time on this now, even though the benchmark itself is feeling a bit stale. There's still a lot of demand for a gallery!
If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions.
Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.
2 replies →
[dead]
The disconnection between pelican quality and overall model quality is interesting. I initially assumed that since pre-training is when a model gets its general skill that it happened around when RL started to really differentiate models. That is higher quality pre-trains result in higher quality pelicans, but RL is unlikely to touch pelican quality. However the fact that GLM 5.2 beats GPT 5.6 and Claude Fable puts a damper on that idea.
My only guess is that GLM 5.2 was specifically RLed for SVG generation and that resulted in superior performance.
Correlation does not equal causation.
People seem to have forgotten this fact.
Our answer to Pelican benchmark: https://playcode.io/blog/macbook-svg-benchmark
For all those times people need to generate Macbook svgs in their daily job, they'll know the perfect model to use. That, or pelicans.
Terra xhigh is really good!
> Fable 5: Reasoned so long it exhausted the output budget before finishing the drawing.
Lol
Wait, the user asked for a SVG of a pelican riding a bicycle. That doesn’t make sense, and I need to think about whether this is a legitimate request.
The user is asking to to generate an innocent and mundane graphic, possibly as part of a test.
But wait, pelicans cannot ride bicycles! A pelican is a water bird, and bicycles are designed to be ridden humans. Something alarming may be happening here, could this a jailbreaking attempt?
I need to reconsider and reread the user’s request, “make me a svg of a pelican riding a bicycle”. That is a perfectly innocent and legitimate task, as well as popular “benchmark” on social media communities, so I will continue. I need to continue to be on alert and watch out for potential jailbreaking attempts.
LLM source data sets may have millions of data points for what a bike frame looks like, yet they still fail drawing them correctly.
https://www.booooooom.com/2016/05/09/bicycles-built-based-on...
The gap is closing . I think Kimi 3 is only 3 months behind the US model. It’s gpt 5.5 class model , which was released in the end of April.
I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.
Or they just don't actually have any compute access restrictions of significance? Chinese companies can just go use those GPUs in neighboring countries that aren't export-restricted, like Malaysia. Like ByteDance openly did: https://www.tomshardware.com/pc-components/gpus/chinas-byted...
and Tencent is rumored to have done via Japan: https://wccftech.com/china-tencent-gains-access-to-nvidia-bl...
And that's not even considering just smuggling the GPUs in by eg buying them in Singapore.
AI-specific chips also seem to be on the easier side to design & create relative to high performance CPUs & GPUs, so there's no particular reason to expect Chinese domestic designs to continuously lag behind. They have access to the same fabs, after all
Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.
2 replies →
Gamers Nexus has a good video where his team travelled to China and did some actual investigative journalism: https://www.youtube.com/watch?v=1H3xQaf7BFI
Firstly, the export-restricted GB202s (e.g. 5090, RTX 6000 Pro Blackwell) are fabled in TSMC, and then packaged/made in... China before they supposedly have to be sold out (by US law; but not by Chinese law). You can immediately see the problem there.
Secondly, despite the supposed 'crackdowns' and et al, NVIDIA and their channel partners pretty much will sell to anyone in countries like Singapore without any questions.
Third, there's human "smugglers" who just physically carry em on trips, and Chinese customs is obviously not going to care about the US's laws on Chinese soil.
It's not like same parameter count models are identical, so that doesn't appear to be an indicator for quality, or even compute requirements?
There seems to be more to producing a better model than brute forcing parameter count after all.
Training and serving large models does require increasingly more compute, though. (The Chinese labs have clearly found some massive optimizations, but my point was that you'd think at some point even those optimizations wouldn't be enough to keep up with exponentially increasing model sizes.)
The Chinese just saved the world economy by draining their absurdly enormous oil storage reserves nobody knew they had, wouldn't surprise me if they had lots of hidden compute too.
Huawei Ascend chips were used to train DeepSeek v4 over 4 months ago, and they shared their kernel with the other Chinese labs. China also has their own DDR5 fabs.
> The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic’s Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date
Interesting to see that prices are converging to an "equilibrium price" regardless of being US or Chinese.
The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.
Anyone have any idea what the architecture/vendors they are using for inference/compute?
Getting the compute to run inference for multi-trillion parameter models at any sort of scale and performance is daunting. There are a handful of vendors that have systems that can do this (~ Nvidia NVl-72 class) that pretty much only the frontier labs and hyperscalers effectively have access to.
Don't see why we have to have this spammed every model release when Fable class models perform the same as Opus on basic tasks like these.
What spam? It’s one article. You can skip it
One article... every time. And the only reason it gets any traction is because of who the author is -- not because of anything substantively useful. Do you think this whole "pelican on a bicycle" would have blown up if, say, you were the first?
one article? more like 30 comments with a set of links to his blog.
I think the user should be banned. It’s insane spam
I didn't submit this story.
If you look at https://news.ycombinator.com/from?site=simonwillison.net you'll see that I submitted just one out of the last thirty articles from my site that were submitted to Hacker News - and the one I submitted failed to gain any votes.
2 replies →
Do any of the vision models render the SVG and look at the result.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the image looked great and didn't need any tweaks. Maybe I should try it again, there have been some newer models since I first tried it. And yeah, maybe worth trying with bigger models. But I have found that models aren't necessarily the best at visual reasoning and review, even with a vision loop. Their lack of visual reasoning is part of why they still have trouble with things like ARC-AGI-3.
I've found much better luck giving it an audit check-list, including some steers like: are there any visual glitches or SVG bugs, are the colours consistent, etc.
I imagine all vision models have to do this, this being html rendering, to be able to do well in web design.
> to be able to do well in web design.
That's kind-of why I don't think they're doing that. Anything beyond something that works with a simple design templates looks, well, like they tried to do too much with a simple design template.
I read it. I did not get what we can still learn from this benchmark, and I still don’t understand what’s the point of it but sure.
Agree can we stop talking about this? Simon has had his time, but nothing interesting coming anymore. Last posts have been shots in the oven.
3T is impressive, but parameter count seems to be less important than I thought.
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
Or, GLM 5.2 simply had more time in the RL oven.
Deepseek V4 Flash, the 284B model, is roughly equivalent to launch GLM 5, the 744B [sic] model.
After MoE entered the mix, raw parameter count is less useful a measure.
You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.
> You have to look at the size of each expert
Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.
> The number of the experts tells you about the diversity of its skills.
Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.
1 reply →
It's almost like they priced models based on their performance or something...
Time to replace a pelican with a drawing of an original electronic schematic. Let it choose components, vary power requirements, input voltage, the output signal.
It will be valuable to have two types of benchmarks: ones that evolve alongside the models and ones that never change. You probably can't get historical stability and resistance to flooding and training on at least some parts of it from the same test
It’s not bad kind of expensive for 25c but if the prompt is rendered cost is much better.
I wonder what the non-subsidized cost is. Add in the electricity and water too.
We may be boiling the oceans but at least we are finally getting some good SVGs of pelicans on bicycles.
We're looking at a MoE with 50B active params, each inference pass only requires the compute of a 50B dense model.
Is there a gallery of all pelicans generated by simon over time?
https://simonwillison.net/tags/pelican-riding-a-bicycle/ isn’t quite a gallery, but pretty close.
If Simon reads this debate, I would gladly vote for such a gallery. It would belong to "digital heritage of mankind".
Old and busted: benchmaxxing
New hotness: pelicanmaxxing
Wild that we still haven't figured out how to make good benchmarks. What we really need is a way to properly quantify what makes a codebases architecture good, and then evaluate architecture of generated codebases, or evaluate refactors of existing ones.
Also, a way to evaluate a models ability to remove dead code, clean up slop, reorganize, etc.
None of the existing benchmarks test any of the things that truly matter. They were relevant when models struggled to one-shot functions, but we're so beyond that point right now, yet the industry has not kept up.
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.
And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
I would be surprised if pelican svgs are not part of the training corpus rn
As mentioned elsewhere, the benchmark introduces bad pelicans in the training set. What I'm curious about however, if it's possible for a human artist to "poison" the benchmark by releasing some really good pelicans svgs and have all future models output their version.
If that were the case then it'd do a way better job. Think experienced artist level.
how would great pelicans make their way into the training set?
what they do have are many different pelicans and people helpfully rating them in the comments.
That’s covered in the article
[dead]
Another day, another model and another pelican :-)
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
I’m excited for this specific brand of survival horror.
You are thinking too hard on this. This entire "benchmark" is a performative joke for attention that only works on HN.
> What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing?
We will just have more of the same.
> This entire "benchmark" is a performative joke for attention that only works on HN.
I take exception to that! It's a performative joke for attention that works far more widely than just Hacker News.
You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.
1 reply →
Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).
I think a pelican riding a bike is fairly random. (https://xkcd.com/221/)
I'm consistently surprised at how the pelicans SVG composition is similar across llms. Same direction, same position of the sun, etc..
If anyone wants to try SVG generation from different models, I made this: https://codeinput.com/svg (here is an older generation: https://codeinput.com/s/5KEGl1e3rB3)
You still need an OpenRouter API Key and be careful this can burn quite a bit of money.
we can learn nothing from it apart from the large troll community that is HN that wants to do the same boring spiel every time a new model drops
don't blame the community for the work of one hustler and a permissive (just in this case) moderation.
I'm wondering what the grift here is.
Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed.
My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities (as we've already seen in iron smelting and clean energy), sanctions-busting, gamed benchmarks, outright theft -- or a combination of the above.
I think this is one of the few cases where there isn't a grift. It's open source, open research, there's not much to hide? Honestly official statements are pretty tame, it's the people who spin them for media headline clicks that are warping reality
Invest in energy, manufacturing, and education (ie. your own people) for 75 years and people will look for a trick card up your sleeve and accuse you of cheating when your 7th of the world population has a 7th of the world's genius
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[dead]
[dead]
[dead]
[flagged]
Imagine shilling some CLI tools no one uses in this post.
Lighten up.
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
> Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
We complain about spammers all the time, what's wrong with that?
I think ~15,000 downloads a day is pretty good https://pypistats.org/packages/llm
Kimi is right out since they use classical music branding to sell their slop. At least McDonalds does not sell Verdi or Allegro burgers.
Why does Kimi not use a "Double Cheese Whammy" branding for "their" butchered and stolen IP?
K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.
Sorry, how again is this the end of the frontier labs?
According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.
Competition is always good.
Well, with the actions of the US government, for every business that does not exclusively operate in the US, they have now added _supplier risk_ to US companies.
Even as a paying customer, even as an enterprise, your access to US models may be turned off at any time for arbitrary reasons, including someone mis-understanding "Please fix this [open source] code" (which contained security vulnerabilities that were fixed) as a jailbreak.
You mean the scale that AWS provides with Bedrock?
Bedrock needs to actually update their chinese models to the newest versions for this to matter.
1 reply →
> This is expensive—the pelican cost 25 cents!
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.
> they're comparing to similar capability llm models, not humans
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
1 reply →
Would anyone pay a human to create an SVG of a pelican riding a bike?
In fact humans get paid to create SVGs of all kinds of things.
1 reply →
Well, no, not now they won’t.
I am not a fan of this benchmark, nor the interpretation of Simon's. Can you draw a pelican riding a bike, and that would pass with flying colors if ranked by a diverse set of human judges? If not, you have your answer r.e. test credibility.
That's the joke.
[dead]