Comment by gwern
10 months ago
It might be doable soon, you're right. But there seems to be a substantial weakness in vision-language-models where they have a bad time with anything involving screenshots, tables, schematics, or visualizations, compared to real-world photographs. (This is also, I'd guess, partially why Claude/Gemini do so badly on Pokemon screenshots without a lot of hand-engineering. Abstract pixel art in a structured UI may be a sort of worst-case scenario for whatever it is they do.) So that makes it hard to do any kind of feedback, never mind letting them try to code interactive visualization stuff autonomously.
A few comments on this thread:
Gwern is correct in his prior quote of how long these articles took. I think 50-200 hours is a pretty good range.
I expect AI assistants could help quite a bit with implementing the interactive diagrams, which was a significant fraction of this time. This is especially true for authors without a background in web development.
However, a huge amount of the editorial time went into other things. This article was a best case scenario for an article not written by the editors themselves. Gabriel is phenomenal and was a delight to work with. The editors didn't write any code for this article that I remember. But we still spent many tens of hours giving feedback on the text and diagrams. You can see some of this in github - e.g. https://github.com/distillpub/post--momentum/issues?q=is%3Ai...
More broadly, we struggled a lot with procedural issues. (We wrote a bit about this here: https://distill.pub/2021/distill-hiatus/ ) In retrospect, I deeply regret trying to run Distill with the expectations of a scientific journal, rather than the freedom of a blog, or wish I'd pushed back more on process. Not only did it occupy enormous amounts of time and energy, but it was just very de-energizing. I wanted to spend my time writing great articles and helping people great articles.
(I was recently reading Thompson & Klein's Abundance, and kept thinking back to my experiences with Distill.)
Huge fan of Distill here (and your personal blog).
> In retrospect, I deeply regret trying to run Distill with the expectations of a scientific journal, rather than the freedom of a blog, or wish I'd pushed back more on process. Not only did it occupy enormous amounts of time and energy, but it was just very de-energizing.
Scientific peer review pretty much always is incredibly draining, and (assuming the initial draft is worth publishing) it rarely adds more than a few percent to the quality of the article. However, newcomers are drowning in a sea of low quality SEO spam (if they bother to search & read blogs at all and don't go straight to their LLMs, which tend to regurgitate the same rubbish). The insistence on scientific peer review created a brand, which to this day allows me to blindly recommend Distill articles to people that I am training or teaching. So I, for one, am incredibly grateful that you went the extra-mile(s).