Comment by AmbroseBierce
15 hours ago
2D animators can still feel safe about their job, I asked it to generate a sprite sheet animation by giving it the final frame of the animation (as a PNG file) and asking in detail what I wanted in the spritesheet, it just gave me mediocre results, I asked for 8 frames and it just repeated a bunch of poses just to reach that number instead of doing what a human would have done with the same request, meaning the in-betweens to make the animation smoother (AKA interpolations)
I’ve been using the same test since Dalle 2. No model has passed it yet.
However, I don’t think 2D animators should feel too safe about their jobs. While these models are bad at creating sprite sheets in one go, there are ways you can use them to create pretty decent sprite sheets.
For example, I’ve had good results by asking for one frame at a time. Also had good results by providing a sprite sheet of a character jumping, and then an image of a new character, and then asking for the same sprite sheet but with the new character.
With local models you can use control net, which is simply speaking, the model trying to adhere to a given wireframe/openpose. Which is more likely to give you an stable result. I have no experience with it, just wanted to point out that there is tooling that is more advanced.
the problem here is that text as the communication interface is not good for this. the model should be reasoning in the pose space (and generally in more geometric spaces), then interpolation and drawing is pretty easy. I think this will happen in some time.
At least until someone decides to fine-tune a general purpose model to the task of animation.
Yeah reading this I was thinking, we've got Qwen-Image-Edit which is an image model with an LLM backbone that takes well to finetuning.
I'd be surprised if you can't get a 80%/20% result in a weekend, and even that probably saves you some time if you're just willing to pick best-of-n results
The person behind www.pixellab.ai has been trying to make a SaaS out of that idea for about 2 years already and it just isn't there, the examples in the homepage are extremely cherry-picked, I bet most of their paying customers just use it as a starting point and then spend hours manually fix the sprites, which may be more than enough value for $12 a month and that's great but what is shows is that we are not as close as one would like to imagine, the "one leg in front of the other at about the same depth and the same color" is still problematic to this day; if most pants in the world had a different color for each leg I bet most of its animation issues would be solved, unfortunately we don't and most of the training data involves single-color pants/legs.
1 reply →
When I tried the same with video models a few months ago by extracting the frames, it was not working so well either.
However, this should be solvable in the near future.
I'm looking forward to making some 2D games.
However, if you ask it to generate eight or 10 frames of a sprite performing a particular action from scratch it gets it pretty spot on. In fact, you can drop them straight into an animator and have near production quality.