When pre-industrial farmers were not in the fields all day they were not idle. They were doing tedious low intensity tasks all day instead of doing laborious tasks all day. A large part of the purpose of Sunday being a day of rest in protestant religions is that if you don't convince people there will be literal hell to pay for not taking a day off they won't take a regular day off and they will work to the point of overwork and reduced output.
Even a naive napkin math estimation of the labor required to keep a 1810s farm house and barn maintained and a few work animals fed and watered should make it clear how much labor needed to be put in before you even get to the farming aspect of it. And it's not like your wife gets to share the labor since she has her hands full with feeding and clothing the family. Pre-industrial farmers did not have the luxury of factory made clothes nor grocery store butter nor a galvanized barn roof nor a tractor.
In the US most of these people were literate and we have enough written accounts that we can accurately reconstruct how they spent their time. It suffices to say that they didn't deforest the northeast and carpet bomb it with stone walls by working 40hr weeks.