Ask HN: Gen model to 1) design characters 2) make a consistent comic with them?

5 days ago

I am looking for a generative model that can make a drawings based on an RPG campaign I had with my friends.

The idea is to create consistent character designs of each character first (say ogre, mage, dragonborn)

and then use those designs for panels that would be based on my inputs about the story (I have notes from a longer campaign).

The models I know / tried are not consistent at all.

They dont store the character design, so each panel is different in style (apart from those model that generate designs ripped off from Studio Ghibli).

Is there any free or paid model that allows that?

6 comments

rvba

Reply
vunderba  5 days ago

You need to build out a baseline style using text-to-image and then always add those images as "style reference" images in future prompts e.g. "Create a four-panel comic of bla bla bla. Use the attached images strictly as stylistic references only".

That will help keep the model on track.

  • rvba  2 days ago

    What model allows that?

    Preferably one that can be self hosted

    • vunderba  2 days ago

      Self-hosting is going to be more difficult. You’ll need to learn how to use Stable Diffusion and some of the common frontends like ComfyUI and Automatic1111 (recommend the NeoForge variant), etc.

      Really strong character consistency is usu. done by fine-tuning a custom character LoRA. If that’s more work than you want to take on now, you could probably get by using ControlNets. The ones you’ll probably want to experiment with first are either `revision` or `reference_only`.

      You’ll also want to start with a solid base model. Since I’m not familiar with the exact type of art you’re going for, it's hard to give advice here. As a first step look through some of the Illustrious and/or Anima models.

      https://civitai.com

      2 replies →