Yes, hard physical work was the main form of "exercise" for most people.

Or you worked until you were told you were done for the day, and no sooner.