Comment by lukev
8 hours ago
What if they’re not pelicanmaxxing, but svgmaxxxing in general?
Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.
Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.
https://www.youtube.com/watch?v=jgYYOUC10aM
reminds me of this Key and Peele skit
Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops.
Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).
Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.
(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)
I've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.
1 reply →
How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good.
AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.
Just today I asked Claude Fable to take an SVG map and add a circle with a 150 mile radius around a specific city, cutting off the circle at the edge of certain boundaries. It came back a minute later with more or less exactly what I wanted, saving me maybe 5 to 10 minutes of photoshop time. It’s almost certainly not perfect, but I didn’t need it to be perfect, just a legible representation for a group of 20ish people.
I used Claude to plan some landscaping, and it gave me SVG diagrams of flower beds showing where to place plants and their approximate mature size. It was pretty useful.
I used it to generate some icons for my app recently, I found it a pleasantly useful feature already in its current state. Before this I would’ve had to close the code editor, open something like Illustrator, use something like a tablet (or struggle with the mouse and keyboard), and export the result. For something like a generic icon for a button, or a simple geometric figure, this always felt too tedious. On the other hand, my apps looked visually impressive before the AI and people valued my artistic ability more. So there’s that… Not every software engineer also had an eye for design. With AI generation, they can bridge that gap.
Great for charts, illustrations etc, can do much more with SVG than pretty pictures. Good spatial understanding in an SVG can be really nice for model to have
AI can generate a fairly satisfactory SVG for a favicon now (programmer art quality at least).
Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.
As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.
Would it be useful if the models were actually good at it?
Vibecoding a SVG based metroidvania as we speak! This is gonna be lit!
The fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.
I use LLMs for 3D CAD design in OpenSCAD.
There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD.
Anecdote I know, but there does seem to be generalization going on here.
If a model can improve at drawing "X doing Y" and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text.
well that holds IF svgmaxxing is 100% "code-writing-maxxing"
...which.. hmm I dunno if they are same or not
No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).
That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.
7 replies →
I don't see why that's true. LLMs don't have to only be good at code-writing.
I've been pretty happy with LLM svg based data plots I've asked, including log scaled axises and histograms. Definitely a first world problem of course.
If they're optimizing for SVG generation, then that's an excellent outcome in my opinion. Vector images shouldn't be "pretty niche".
Please it's almost my primary bench for drafting understanding.
I'm almost about to post that xkcd 810
Gemini have absolutely been SVGmaxxing. They've openly talked about it.
Then it's great. A year ago I couldn't get any AI to draw a simple company logo in SVG or even convert from raster. No doubt the Pelicans put pressure on the labs to fix the awful SVG situation. Now I can even make a decent Peli in 3D.
Not an expert on this, so won’t speculate regarding what traits would create robustness specifically attributed to SVG visual representation capability, but felt you might find this paper on reasoning models trained on physical world video data becoming better at general reasoning interesting:
https://arxiv.org/abs/2210.05359
Addressed in the article, in case you're curious.
Addressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.
Well, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.)
That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)
Then they’re legitimately getting better at svg which is a valuable tool.
I think svg is a balanced test because of the level of indirection and the required 'conceptualization' of physical elements then expressed through code.
Funnily enough, not that niche, because I have tried many times to do it as part of a wider project.
SVG are really useful you can create images that don't have the AI look
I agree, other formats, both textual and binary should be tested.
> What if they’re not pelicanmaxxing, but svgmaxxxing in general?
Mission fucking accomplished. https://xkcd.com/810/