Comment by toast0

> Look at the farms that have the houses of that era standing on them and you'll soon notice that they are all mansions.

TLDR: survivorship

The typically large farms with nice houses were making reasonable money, and in a lot of places, only the house remains of the farm. My old neighborhood was a large farm, subdived into about 1000 postage stamp lots around 1900; the owner's house got a slightly larger lot and stuck around as your mansion.

The small farms that were within the means of more people tended to have shanty houses and those have not persisted. If the farm is still a farm, it's likely been subsumed into a larger plot.