Comment by simonw
3 days ago
Yeah this was pretty surprising. For almost every model I've run the pelican against the first attempt was at least recognizable enough that I didn't feel like the model needed a second shot.
It's always a roll of a dice, but it's surprising that the dice rolls so infrequently come up bad, yet Opus 5 rolled a bad pelican this one time.
I suspect it's just a freak occurrence. I rolled a few more and they were all fine, I think Opus 5 just got unlucky.
What I think is more alarming is that Claude's advice for prompting Opus 5 says that Opus will verify its own work [1]:
> Claude Opus 5 verifies its own work without being told to. If your prompt contains explicit verification instructions ("include a final verification step for any non-trivial task," "use a subagent to verify"), remove them: instructions like these cause over-verification on Claude Opus 5, and removing them reduces wasted tokens with no loss in quality.
My own experience over the past ~20 hours hasn't been great either, with Opus producing sloppy mockups (e.g. buttons overflowing past cards) without doing any of the purported verification.
[1] https://platform.claude.com/docs/en/build-with-claude/prompt...
The way I run the pelican benchmark prevents it from checking its own work.
It gets one API call to return an SVG.
If I ran the benchmark in Claude Code or a similar harness it could render the SVG as an image, look at what it created, then make tweaks to it.
Ah, thanks for clarifying!