Comment by Avamander

Do you have a stereo setup?

I suspect downmixes to stereo and poor builtin speakers might be heavily contributing to the issue you describe. Anecdotally, I have not encountered this issue after I added a center channel.

Nor do I have any issues with the loudness being inconsistent between scenes. I suspect that might be an another thing introduced by downmixing. All the surround channels are "squished" into stereo, making the result louder than it would have otherwise been.