Comment by Cthulhu_

Exactly; consider that only a percentage of users will actually be sitting at a desktop system using your application. Most people will visit your website on a mobile device, and a percentage of that will be in weird lighting conditions trying to operate the thing with just one hand. As just an example, and that's the people considered abled - anything that isn't the hip startup of the week aimed at 15-25 year olds will need to consider e.g. eyesight issues on top of everything else.

Anyway, it's not even an option anymore, under the European Accessibility Act, government and (since last year) public services (websites, apps, appliances, etc) all need to be accessible; for websites this means implementing WCAG 2.something to level AA, which includes things like minimum contrast requirements, captioned videos, and increasing font size without losing information.

And it's often considered an afterthought because by and large, designers and developers are relatively young, 20-40 some year olds working on Macbooks with 4K displays. Some may be wearing glasses but broadly speaking that's the extent of their disabilities.