There is no problem with low birth rates in the long term. There are, however, massive problems during the many decades when the population pyramid is inverted and there are 4 or 5 retirees for every child, a ratio which Japan is about to hit and where South Korea will be in a couple of decades as well.

As for why this is a problem, Japan's pension system is already insolvent and the country is starting to experience massive manpower shortages in sectors like nursing, agriculture and construction. Sometime last year Japan passed the point where the amount of money & effort needed to maintain existing infrastructure exceeded the country's capability to build it, and the depopulation and dilapidation of Japan's second/third tier cities and the countryside is already striking and poised to accelerate rapidly as villages, towns and cities can no longer fund their basic obligations.