This is clearly not true. They didn't have modern leisure activities, but they had a vast array of activities to keep them from getting bored when they weren't working or doing the arduous, nearly continuous preparation of meals.
> Common people had almost no sports, no games (beyond precursors to Bocce or backgammon), no literature!
No literature, sure, because they were illiterate (and it took the invention of the printing press to create a market for leisure books).
But no games/sports? How about boules, bowling, prisoners' bars, blind man's bluff, table games (chess, checkers, backgammon, alquerque, three-in-a-row, mill, the fox and geese, tablut), dice, card games, variations on golf, hand-ball, kick the can, cockfighting, cow-tipping, bull-baiting, a form of rugby, wrestling, fencing, racing, and an innumerable array of local games often surrounding festivals with cultural/spiritual significance? They also did activities like swimming, fishing, hunting, playing music, singing, story telling, dancing, even ice skating.
I'm tired from just listing them all!
The most common leisure activity for men was probably drinking in the tavern. This shouldn't be understated; this took up a lot of time. And it wasn't because they had nothing else to do, it's because drinking and socializing is often preferable to the above activities, even today. A lot of people today don't play any games at all, but spend hours every day sitting around shooting the shit over cans of Bud.
I'm not as familiar with womens' lives, but I imagine they had more responsibilities and less leisure time. Cooking, cleaning (such as it was), sewing/needlepoint, and raising children all takes considerable time, so they mightn't have had as much time for leisure. A lot of the above activities were also intended for men.