Comment by suprjami
12 hours ago
> The Astra pelicans are much better.
It's absolutely in the training data now.
Time to retire this (previously mildly amusing) benchmark as frontier models are now pelicanmaxxed.
12 hours ago
> The Astra pelicans are much better.
It's absolutely in the training data now.
Time to retire this (previously mildly amusing) benchmark as frontier models are now pelicanmaxxed.
What training data? Is there a big library of excellent SVG pelicans to train of?
Yes, all the people talking about this benchmark since OP created it.
Talking, sure. But have they created any good pelicans to train on? Pelicanmaxxing is clearly not a thing, which can be demonstrated by asking it for something else, or migrating to another domain (voxels, 3D modeling, and other things an LLM is supposed to be bad at, plus an unlikely combination of concepts). Looking at how relatively good for an LLM Astra is at Blender, it's obvious it was either purposefully trained for better spatial performance, or just generalizes better.
I actually think the pelican and similar tests are much less useful than it seems, but not because of pelicanmaxxing. They are supposed to serve as a vibe check of model's out-of-distribution performance, but do nothing to disentangle the generalization and memorization, which is the hard part. The combination of both is still useful though, and if you look at pelicans over time you'll see their quality is more or less correlated with that, with the exception of models specifically trained to produce vector graphics and 2D layouts.