At least in crop agriculture the work tends to be concentrated during planting and harvest so there is a big period in between with nothing to do for most of the people involved in planting and harvest. The labor for at least some of the Egyptian pyramids were idle farmers who had free-time available for hire, iirc.