Comment by rdtsc
15 days ago
Me in 2025 https://news.ycombinator.com/item?id=44759781
> If it's on HN and is a meme at this point, it will end up in the training set.
> It's kind of fun to imagine that there is an intern in every AI company furiously trying to get nice looking svg pelicans on bicycles.
Google listened and assigned a few interns, I guess, for the task
That's exactly what I was thinking when I saw https://www.assbench.com/ posted a while back: "oh no, this is going to be somebody's job".
NGL, I think this might be my favorite benchmark, ha. While the Chinese labs asses are a bit off, I certainly have big hopes for their next versions. Their hearts are certainly in the right place.
It's the smaller training data /s
Unfortunately, for me the renderings crash or hang in both Firefox for Android and Chromium/Vanadium on a Pixel 10 Pro. :\
Works fine on Android + Firefox for me.
omg that's hilarious
prompt is to detailed (llm written?) better test of model creativity is to not constrain prompt
SVG is useful for actual tasks, not just looking good in a silly "benchmark"
Beyond SVG, these things have been RL'd on 3d reconstruction (which carries over), and I expect given the public reaction to recent frontier advancements in 3d capabilities that this will get allocated a larger share of the post training budget in the future.
What I'd kill for would be an app where you take a pic of scribbles and drawings on a whiteboard/notebook and the app turns it into SVGs you can then continue to manipulate digitally on PC/tablet.
Why would you kill for this? Why not just generate svgs with any editing software?
1 reply →
Inkscape and https://github.com/visioncortex/vtracer can both do this.
1 reply →
ChatGPT does that decently well
7 replies →
Of course it would be nice if google actually added SVG support to google slides...
We all said it: https://news.ycombinator.com/item?id=46978726 (8 months ago)
> How many pelican riding bicycle SVGs were there before this test existed? What if the training data is being polluted with all these wonky results...
But Simon has demonstrated before that asking for any animal doing something silly will still get similar results from the same model.
I recently learned that there is an animated pelican riding a bicycle in 1988's Who Framed Roger Rabbit! https://simonwillison.net/2026/Sep/18/the-creative-spirit-of...
Do you think this might be a kind of subconscious origin of your idea? A second of a move you saw in your early life / teens stayed with you as an example of an absurd thing to draw?
1 reply →
It's still clearly not a part of any model's training set in the sense of pelicanmaxxing in particular. It's either a capability to generate arbitrary SVGs (or even something broader), or no specific training at all.
Yet, they're all almost exactly the same in style and composition.
Certainly not because all labs are pelicanmaxxing on the same exact image. Most patterns are similar everywhere. That's the result of training on the same datasets and scraped web contaminated with other models' outputs, combined with similar training methods and resulting mode collapse. Try if e.g. dolphins knitting socks in spacesuits are similar on all/most models.
4 replies →
Certainly true but I think the question is “does targeting the SVG pelican mean that the quality of all SVG output improves?” If so it’s a worthwhile thing to target.
It's a little bit like any benchmark, especially microbenchmarks. It is useful as far as it reflects the broader use cases, but the risk of overfitting to the specific task and not the broader ones is very high.
https://xkcd.com/2899/
They said they are training for svg not pelican svg specifically
Sadly, this is all that managers know to do: take a metric and try to optimize the heck out of it. They don't care about research, technology, "don't be evil", etc. And Google is run by managers now.
Expect Google models to be benchmaxxed as much as possible, but don't expect commensurate performance on real-world tasks.
>> this is all that managers know to do: take a metric and try to optimize the heck out of it. They don't care about research, technology
This is not true. It is needless negative hyperbole that IMHO doesn't belong on HN.
I'm speaking from 30 years experience in the Valley, having worked at several big companies.
While this might be overly cynical, it's also a powerful validation of the exact feedback loop that you want to have in an organization.
If what you say is true, then the next set of benchmarks will be written to illuminate the true reflection of real-world tasks. Then those get benchmaxxed so managers can show "numbers go up."
Repeat enough times, and you get materially and genuinely improved products.
The burden is on the press, outside researchers, and general tech industry to both write and care about the metrics that matter.
There's just one problem - culture eats strategy for breakfast. You cannot govern culture or improve culture with metrics.
In large companies (Fortune 1000 etc.), you don't have much of a choice, the culture was eroded long ago, and corporate governance by metrics is a lesser of evils. But thankfully, not all companies are there yet.
For sure, they saw your comment and got some interns on it.
Maybe so, and yet it's still not enough -- the pelicans all have flaws.
They look remarkably good, sure -- but not perfect.
I end up getting SVG output for all kinds of things, and never once have I asked for, or gotten, a pelican.