Comment by littlestymaar

That's what LoRAs are for.

And small models are also much easier to fine tune than large ones.