Comment by simonw
2 days ago
Absolutely the best pelican I've seen from a model that runs on my laptop: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Bicycle is the right shape. Pelican beak is excellent. Nice background. Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.
(No chain on this bicycle though - in the reasoning trace it says "already chainstay... skip chain detail; maybe a small chainring.")
I ran that on an M5 Max MacBook Pro using LM Studio and their 17GB GGUF: https://lmstudio.ai/models/qwen3.8
It took 21 minutes(!) and used 22,276 reasoning tokens to produce 3,223 tokens of output.
(For the "they're training on your benchmark now" crowd, all of that cheating didn't prevent it from spending 20 minutes thinking about the task first! You can see the reasoning trace in the link I shared.)
For comparison, here's one I got from qwen3.8-2.4t-a95b on OpenRouter, which is pleasingly animated: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://tools.simonwillison.net/markdown-svg-renderer.html#u...
Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right.
It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.
Have you seen what happens when you ask 500 humans to draw a bicycle? No pelican, nobody riding it, just the bike.
https://foerstel.com/memory-bikes/
I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks.
Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it.
*Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.
2 replies →
Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.
It is interesting that it went with something like:
"pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican"
I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.
Damn That's very good
>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.
How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.
Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other.
The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all.
Also:
> Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.
Gemini is worse
Holy crap that is above and beyond
But, not animated like the Qwen 2.4T one you showed, so minus points for that
Animation was not requested.
Totally agree. I actually didn't know you already got to it and I ran it on mine. It even added a scarf lol
https://imgur.com/a/IgYKnmk
Took almost 90 minutes on my Threadripper Pro, and feels well worth it.
28k tokens from the Q8 model
And yours got the chain!
What's the result if you ask it for a "pelican equipment case" ? I've been trying the anti-bicycle pelican on some LLMs and the results are much more varied than the bicycle prompt. Some have very different ideas of pelicans (you'll get a small case with one DSLR camera in it, or a long rifle case, etc). You'll also get cases that are isometric view, or flat plane view from the front, or open or closed.
It did a pretty outstanding job on that: https://tools.simonwillison.net/markdown-svg-renderer.html#u...
This took 56 minutes! - but it didn't have exclusive use of LM Studio, I ran a few other test prompts while it was working.
That is pretty good! It's interesting it chose orange, because that's actually a somewhat rare color for pelican cases. I would say 95% of the ones going around the world right now are either black or FDE (flat dark earth) tan colored. There are some orange ones, for sure, if you google image search "orange pelican case", people use them for emergency/first aid supplies or electronics. But only the smallest ones are available in orange.
It is also worth noting that it shaded the pelican case like it was a product photo taken in a studio, there's a virtual light source above and to the top left corner of it, then shading deeper color of orange as the body area progresses towards the lower right corner.
7 replies →
Not too bad, on my 5090 I got ~160tokens/sec and it took 3m20s to generate a pelican using Ninfer which someone else mentioned. I don't know why I didn't get any sweet sweet seagulls in my background, even on xhigh reasoning effort.
https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632a...
I've done a few variations, I've been impressed with all of them. My favourite so far has been "Generate an SVG of a turtle flying a kite", result: https://imgur.com/a/bdKJPV4.
Some will say conflating flying and flying a kite is bad, but I like it.
> Bicycle is the right shape. Pelican beak is excellent. Nice background.
Relative to other results I agree. But on an absolute measure, there is not a single element in the current bicycle that is real-world accurate and many elements are omitted or non-physical (eg, the transparent seat tube top, entire lack of a head tube).
Consider a series of followup benchmarks.
With a fresh context of the LLM under test, ask it to generate a list of findings for how the pelican-on-a-bicycle SVG that was produced is inaccurate compared what the real world scene might appear, accepting for the limitations of SVG as a medium. Then, feed back the list of findings to the original context for a second try. The benchmark can stop here by humans looking at the result and forming their own conclusion.
Next phase is to repeat the analysis phase using the 2nd context to determine what findings were satisfied and what new inaccuracies are found. These two differences can form a second benchmark.
Last phase is to iterate with the goal to drive the number of findings to zero.
Have you thought about running a second tier of the Pelican benchmark where you see which model makes best pelican on lowest or no reasoning settings? I think that'd be pretty interesting and might help highlight which models have a baseline capability - even within those I'd imagine the token usage would vary wildly and might give some indications on verbosity as well.
I was just updating a Proxmox cluster from v8 to v9 and had time on my hands and 15-30 mins after the models appeared for general download I had an initial pelican effort in .svg format.
I switch off reasoning by default and for starters I ran 3.8 the same way as 3.6 on vllm on a DGX Spark box. Open web UI prompted.
My word! It looks rather decent. Much better than previous efforts.
Using a better quant Q6_K (Unsloth's) compared to your quant Q4_K_M (LM Studio's) yields this: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
Chains exist. Red scarf is proactively added. ("Maybe a scarf blowing in the wind for charm!) No hands/wings, though.
Generated 30.2k tokens in total and took 52 mins on M5 Pro in low power mode (it will possibly take less than half of that in auto energy mode).
It's definitely trained on generating SVGs. Not that they trained on the pelican prompt itself, but probably a more general RL step that has it generate SVG and a stronger vision model judges it.
It's up to you if that counts as benchmaxxed.
Generation of SVGs is (obviously) a generally useful task.
Even with "low" thinking effort and Q4 it makes reasonable pelican, only 4k tokens and half a minute on 4090: https://imgur.com/a/yd5rNCp
I asked it to make an animated svg of a pelican doing tricks on a bmx bike and it did. I then asked it to turn that into a browser game. This model fucks.
this sort of test surely only works when it's a brand new concept/prompt. like the exact same models trained newer data will surely get better at putting this together over time
Try it out with your own prompt, I suspect you'll be surprised.
Ask it to do a zebra on a jetski to escape the benchmax.
what type of laptop runs something like that? using mac or egpu?
If I'm doing the math right, that's like 17 t/s? I haven't played with Qwen 3.8 yet, but that seems really slow for a 27B on an M5 Max with sufficient RAM to hold it in memory.
See, like my comment before. We don’t need the biggest/highest/frotiest model.
I think the model got trained only on pelican to get there
We should start asking it for an Albatross instead.