> What was the remaining risk to the US?
Japan never posed serious risk to the US in the first place.
> That's one of the main things that makes it feel hard to justify further mass killing of civilians.
Japan had occupied much of East Asia including large parts of China and was conducting absolutely brutal campaigns of repression.
> What if Japan has just never surrendered? Would that have been justification to drop an atomic bomb on every civilian population center one after another?
The justification is that as soon as you stop containing them in their island. They would instantly start to go on the offensive again.
Long term containment was considered by the US Navy as an alternative, but it would hardly be less brutal then nukes.
What would you have the US do, simply say, well I guess you learn your lessen and go home. Leave Japan to reconquer East Asia again after a few years?
The nukes were worthwhile to try because they had the capability of totally shifting the political conversation inside of Japan, and they did.
Each attack and action had to be individually considered one after another. If nukes would not have shifted to strategic calculation, potentially another policy would have been attempted.