Comment by SpicyLemonZest

Again, there's a very clear user control. On the email visibility settings page, there's a toggle named "Allow connection to export emails". The 5% of contacts who had an email in your data dump are the 5% of people who've chosen to let you export their email by enabling this toggle.