Comment by simonw

That's exactly what I was thinking when I saw https://www.assbench.com/ posted a while back: "oh no, this is going to be somebody's job".

SVG is useful for actual tasks, not just looking good in a silly "benchmark"

We all said it: https://news.ycombinator.com/item?id=46978726 (8 months ago)

> How many pelican riding bicycle SVGs were there before this test existed? What if the training data is being polluted with all these wonky results...

But Simon has demonstrated before that asking for any animal doing something silly will still get similar results from the same model.

It's still clearly not a part of any model's training set in the sense of pelicanmaxxing in particular. It's either a capability to generate arbitrary SVGs (or even something broader), or no specific training at all.

Certainly true but I think the question is “does targeting the SVG pelican mean that the quality of all SVG output improves?” If so it’s a worthwhile thing to target.

They said they are training for svg not pelican svg specifically

Sadly, this is all that managers know to do: take a metric and try to optimize the heck out of it. They don't care about research, technology, "don't be evil", etc. And Google is run by managers now.

Expect Google models to be benchmaxxed as much as possible, but don't expect commensurate performance on real-world tasks.

For sure, they saw your comment and got some interns on it.

Maybe so, and yet it's still not enough -- the pelicans all have flaws.

They look remarkably good, sure -- but not perfect.

I end up getting SVG output for all kinds of things, and never once have I asked for, or gotten, a pelican.

Is it me or is there maybe some carryover happening where the pelicans are portrayed as expending different degrees of effort too? Like, the high effort pelican has a determined look on his face.

Is the prompt context telling the models what their effort level is, and the ambient presence of that concept of ‘high effort’ in the context window is leaking through and affecting the portrayal?

That's absolutely not true. Zero rake forks are totally fine and very common on BMX bikes for barspins.

You can also ride a negative rake fork. They are less stable, yes, but a far cry from not actually rideable, cf https://www.instagram.com/yourforkisonbackwards/

I love this one https://www.gianlucagimini.it/portfolio-item/velocipedia/

People drawing bikes, scroll down to see they having made renders of all the drawings. Hilarious

> Given LLMs are generally good at basic maths, im surprised they get this wrong.

1. Modeling and analyzing spinning, three-dimensional objects goes far beyond "basic maths".

2. The prompt was for an SVG of a realistically impossible situation (pelican riding a bicycle), which together already implies a stylized depiction. Why on Earth would you expect it to generate a 3D model of a bike, do a physics simulation on it, iterate until it satisfies certain contraints which are only hinted at in the prompt, and then project it down to a simplified 2D cartoon drawing? That's not how LLMs work, and that's not how we want them to work.

I feel like perhaps this is a bit too nuanced of a critique. While I do see what you are saying on many images of bicycles, there are also many that that I do not see it on. Close to half of them, in a quick Google image search, have forks appear straight and the axle is dead center on the ends of them. If there is something there, I am not seeing it even when trying to specifically look for it.

I'll also note that, using a piece of paper as a straight edge against my screen, there is a very slight forward curve on the low and a bit more on the high responses. Medium does appear to be perfectly straight.

Yeah but if 99.9% of people would get the bike totally wrong, why does it matter if the LLM doesn't get it perfect.

You're nit-picking on a detail no one actually cares about.

If that's the case, how do bikes like this work?

https://www.walmart.com/ip/Huffy-26-inch-Rock-Creek-Men-s-Mo...

(apologies if I'm misunderstanding).

He deserves a full parliament

Pouchbench. There's an RLVR grader that performs pouch detection and area estimation. Bigger pouches, bigger reward. Soon, all will be pouch.

A pelican with an empty pouch would have no need for a bicycle to get home.

Well, a photorealistic pelican would generally be in a raster-graphics format rather than an SVG, so yes, cartooniness is expected.

Pouchmaxxing

The SVG is cartoonish, and to be honest, most of human-made SVG art is cartoonish or stylized. It's not a format with realistic expectations.

Agreed. It's a fun cultural tidbit but not really a useful benchmark for model capabilities.

There's plenty of highly complex benchmarks for this stuff. Simon doesn't need to do that.

Definitely the best looking on a shirt. Already had my agents add the Gemini 4 Argon pelican on a bicycle as a tshirt on pdoomers https://pdoomers.com/products/pelican-on-a-bicycle/?model=ge...

> public misconception about how current AI works

Another case where allowing oneself to anthropomorphise LLMs a little yields better intuitions.

People don't get more skilled by raw reasoning, either. They improve their skills with better textbooks (pre-training data), more practice exercises (RL data), and generally goofing around (self-play).

Why? Don’t you think the steady improvement in pelican bicycling SVG art is itself a worthy field of endeavor for LLMs to be optimized for?

Yes, Simon has made pelicans a benchmark and now models have grown very good at rendering them. Is there a reason to believe that this isn’t manifesting as useful skill in general in SVG generation?

If a result is suspiciously good I run a spot check with one of my many other SVG things.

Here's 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars' for Gemini 4 Argon: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Some other models for comparison: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

That's true.

I've noticed that chatgpt is working harder. But still I prefer Claude, it just feels nicer to me.

Check the "high" thinking output again. For what it's worth, it's there.

However, how the front fork rake affect stability is I believe a little complex. I doubt this old article is the last word on the subject, but it's a great read nonetheless, and note especially the ultra-stability of the reversed fork bike!

https://physicstoday.aip.org/features/from-the-archives-the-...

The models that are really good at 3D modeling tend to also be really good at pelicans. I think the skills are strongly related - it's about having good 2D (and now 3D) spatial awareness.

The test is always going to be hard though, because it's an absurd image to ask for.

UPDATE: I like this theory about using an agentic harness that can render and iterate better: https://news.ycombinator.com/item?id=50073879#50074395

Models seem to have very interesting blind spots. While https://assbench.com/ may seem like a joke, the results are very illuminating. One might expect far better results given the detailed prompt, allowance for render checks and iteration, the number of human figures the models are trained on, and clarity of tool use (threejs).

However, result quality really drops off once you get out of the top 5 or so.

(NSFW note: all buttocks are covered by fairly conservative bikini bottoms and show no genitals, bulges, etc.)

The pelicans are one-shot with not tools: Think a bunch, write a bunch of SVG, done.

The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.

"Pelican on bicycle in SVG" is an adversarial example on multiple levels - out of distribution, ill choose of tool for the task, and anatomical impossibility of real pelicans using typical bycicles.

A lot depends on what the models were specifically trained for using reinforcement learning. Modern models are generally OK and everything, but they're only really good at the things they have extensive RL for.

> Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?

Are you sure they're not fetching free asset packs for that?

Have you tried editing SVG by hand?

That of course depends on what you want the model to do. If you are into artistic SVG rendering, it's still very relevant.

shun the heretic!

Buddy, there's a Pelican riding it. No real cloud is like that, no real sea is like that, no real gulls are like that. And that's fine because it's a cartoon.

It attracts at least one comment like yours on every single thread, so I guess the main purpose is to kick off a race to see who can post that comment first.

I enjoy it. Many readers here look specifically for Simon's pelican comment when a new model launches

Memes and self-promotion.

It's merely become part of the culture of new model drops even as it's become a bit less useful as a diagnostic.

low version doesn't have a chain.

Medium and high are fixies apparently, (or they have a worm gear, although the hub doesn't look big enough for that), since neither has a derailleur. That would also explain the lack of brakes at least.

Low is still obviously failing the "test". It's pedals are not actually attached to the bike either.

Medium has a handlebar that goes through the pelican's beak. So I'd say also failing.

For the high, the only nitpick would be that the bell isn't quite on the handlebars and the stripes on his scarf don't line up correctly.

It seems to be a deadpan inside joke around here that everyone pretends to care about the pelican test.