Comment by amluto

One possibility (pure speculation) is a bad antialiasing filter. The Nyquist frequency at 44.1ksps is 22.05kHz, which is only ~10% above the audible band. This means that you need a rather sharp filter both when downmixing and when playing to avoid potentially audible aliasing into the audible band or attenuation within the audible band.

If you look at a site like audiosciencereview.com and pull up measurements of a DAC or ADC, you can find graphs of the antialiasing filter response. Some are great and some are not.

One could think of 16/44.1 PCM as being a codec that is potentially perfect but requiring some degree of care to encode and decode correctly.