Comment by FarmerPotato

One way to say this is "present company excluded" as when saying: culpability is general in the population, but not to accuse those you're speaking to.