Comment by orbital-decay

Talking, sure. But have they created any good pelicans to train on? Pelicanmaxxing is clearly not a thing, which can be demonstrated by asking it for something else, or migrating to another domain (voxels, 3D modeling, and other things an LLM is supposed to be bad at, plus an unlikely combination of concepts). Looking at how relatively good for an LLM Astra is at Blender, it's obvious it was either purposefully trained for better spatial performance, or just generalizes better.

I actually think the pelican and similar tests are much less useful than it seems, but not because of pelicanmaxxing. They are supposed to serve as a vibe check of model's out-of-distribution performance, but do nothing to disentangle the generalization and memorization, which is the hard part. The combination of both is still useful though, and if you look at pelicans over time you'll see their quality is more or less correlated with that, with the exception of models specifically trained to produce vector graphics and 2D layouts.