The overall average per person has declined (though not really since the 80s). However, in the same time frame, the proportion of women in formal work went from 35% to 60% [1]. So the story for men and women looks very different. Note also that for women this is a shift from domestic labor to formal labor, so it's probably not much of an increase in hours worked even though it looks like that on paper. However, in the same time frame, I believe men spend a bit more time on housework than they used to, which is not included in formal hours worked.
The point of all this is simply that it is difficult to make conclusions about labor market trends from average annual hours worked.