>I don't see why fertility rates wouldn't change with time.
Of course they'll change with time. They'll go down further. Not what you mean, you somehow believe they'll go back up... but without bothering to come up with an explanation for why they'd do that. Maybe because it's emotionally comfortable to imagine it so, or convenient to your argument.
In reality, low fertility rates cause economic stagnation that most believe to be the cause of even lower fertility rates. It will spiral.