Comment by bilsbie
15 days ago
Regarding the pelican test. Maybe someone will know this.
Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?
15 days ago
Regarding the pelican test. Maybe someone will know this.
Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?
The models that are really good at 3D modeling tend to also be really good at pelicans. I think the skills are strongly related - it's about having good 2D (and now 3D) spatial awareness.
The test is always going to be hard though, because it's an absurd image to ask for.
UPDATE: I like this theory about using an agentic harness that can render and iterate better: https://news.ycombinator.com/item?id=50073879#50074395
> it's about having good 2D (and now 3D) spatial awareness.
this is a really profound insight
Models seem to have very interesting blind spots. While https://assbench.com/ may seem like a joke, the results are very illuminating. One might expect far better results given the detailed prompt, allowance for render checks and iteration, the number of human figures the models are trained on, and clarity of tool use (threejs).
However, result quality really drops off once you get out of the top 5 or so.
(NSFW note: all buttocks are covered by fairly conservative bikini bottoms and show no genitals, bulges, etc.)
The pelicans are one-shot with not tools: Think a bunch, write a bunch of SVG, done.
The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.
Yeah, that's a good point. My basic pelican test doesn't allow the model to render their work and iterate on it. The 3D models all come from agent systems that are rendering with Blender or screenshotting with Playwright along the way.
Have you looked at assbench.com? Even with threeJS rendering, screenshotting with Playwright, and iteration (check the prompt) some of the models are shockingly bad. I wonder whether the rendering quality is affected by the requirement for skin physics on click; if so, it raises a valid question about models creating pretty pictures but break when interactivity is required.
(As I said in a separate comment above, results are all bikini-clad and not revealing. However, whether it is NSFW is up to you/your work policies.)
I’d really like to see what each model comes up with if given a basic harness to render and iterate. They’ve all gotten pretty good at basic one shot pelicans, but seeing what they do when given a direct goal, and even getting a time lapse of each image as they edit and adjust would be really fun.
1 reply →
If you allow rendering and iteration on the basic pelican test, does it improve the final output?
1 reply →
"Pelican on bicycle in SVG" is an adversarial example on multiple levels - out of distribution, ill choose of tool for the task, and anatomical impossibility of real pelicans using typical bycicles.
But often these models mess up drawing even just the bicycle part.
I remember a submission a few years ago about a guy asking others to sketch bikes. Most of the time the drawn bikes wouldn't be rideable.
https://news.ycombinator.com/item?id=17699017
To be fair, humans can’t draw a bicycle either.
https://www.wired.com/2016/04/can-draw-bikes-memory-definite...
A lot depends on what the models were specifically trained for using reinforcement learning. Modern models are generally OK and everything, but they're only really good at the things they have extensive RL for.
> Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?
Are you sure they're not fetching free asset packs for that?
Yes. I've been doing a bunch of 3D experiments with recent models, you can check the session logs and see if they've retrieved anything. They're building the 3D models directly in code.
I even built an environment that allows execution of Python code to render with Blender but allows no other interactions (including networking, I sandbox that off). They can create good 3D models just fine - here's a Faberge Egg by Gemini 4 Argon: https://tools.simonwillison.net/blender-viewer?url=https%3A%...
Oof (no pun intended), that's a huge step-change in competence over last time I tried this.
Thanks for doing all this research!
Have you tried editing SVG by hand?