Comment by libraryofbabel

Although interestingly, as that wikipedia article also points out, people did go down into the 40s quite a bit during the Age of Sail (the famous clipper route), because the strong prevailing winds meant it was the fastest way to get around the world. This comes up quite often in the Aubrey-Maturin novels by Patrick O'Brian.

If you're not a sailing ship, you don't benefit from the winds, so those latitudes are pretty empty nowadays.