Comment by btown

> The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.

In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal before being told to continue, and every few tool call outputs add comment lines that refer to SVGs of pelicans (and, perhaps, how a møøse bit my sister once). And, at the end, once it's 800k tokens deep into context, it's asked to produce an SVG of a pelican and is evaluated against both the pelican and the completion and efficiency of the task.

You're only as good as your ability to solve problems in the midst of an SVG pelican attack.