Plus it is probably a good thing population will start dropping.
The much larger worry should be Climate Change, a dropping population can only help Climate Change in the long run. But right now, due to how we all live, we are heading into a whole lot of hurt due to Climate Change. Far more "hurt" than the population falling.
Also, worried about population dropping ? Wait to see how fast it drops when Countries start massive wars due to dwindling resources.
EDIT: want an example of the Impact of population dripping ? Look at Europe during the Plague in the 1300s(?). What happened was the rich had a hard time finding labor, so they had to start paying people a lot more for their work. To me, that is the big fear, the rich may have to start paying more.