Comment by Khoth

For the other side there's A. K. Gavrilov's "Techniques of reading in classical antiquity", not (legally) freely available but quoted in various places such as https://kiwihellenist.blogspot.com/2016/02/reading-silently-...

The argument for the Ambrose case is that the surprise isn't him reading silently, it's him reading silently while he has guests. (and elsewhere in the text Augustine mentions himself reading silently by himself in a way that suggests it's entirely normal)