A lot of it is explained by rising marriage ages. Women lose the ability to have children at around 40, so the later they get married, the less chance they have of having children, even if all other things are equal. E.g. if we assume an even chance of having children per year, then we'd expect an increase of the average marriage age from 32 to 36 to halve the birth rate (as in the former there are 8 years in which the woman could give birth, while in the latter there are only 4).

This means that even if people's biological fertility and desire to have children didn't change at all, people delaying marriage to later in life would still potentially result in a big drop in the number of children born.