Comment by amelius

It's probably because touching the outer ring of the BNC connector (which is at ground) can become dangerous, yes.

So the best way to go about it is to use an isolated differential probe. Or use an isolation transformer to float your device under test, which is always a good idea.