Comment by pixl97

Models don't have to follow instructions, but during RLHF that is one of the things they are scored on so a premise of the idea is part of the model, but it's also balanced on accomplishing the end goal. They model may determine, correctly or incorrectly, that your rules suck and do what it thinks is best.