Comment by Zak

The linked project is about removing censorship from open-weight models people can run on their own hardware, and your comment addresses incidents involving LLM-based consumer products.

Sure, products like character.ai and ChatGPT should be designed to avoid giving harmful advice or encouraging the user to form emotional attachments to the model. It may be impossible to build a product like character.ai without encouraging that behavior, in which case I'm inclined to think the product should not be built at all.