← Back to context

Comment by simonw

2 days ago

For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://tools.simonwillison.net/markdown-svg-renderer.html#u...

13 comments

simonw

Reply
lwansbrough  2 days ago

Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right.

It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.

  • fwipsy  2 days ago

    I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks.

    Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it.

    *Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.

  • fragmede  2 days ago

    Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.

walrus01  2 days ago

It is interesting that it went with something like:

"pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican"

I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.

zkmon  2 days ago

>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare.

How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.

  • anakaine  2 days ago

    Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other.

    The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all.

    Also:

    > Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.