The fact that we don't understand how neural nets work despite the excellent results makes an AI apocalypse more likely, not less. This means that if we ever create strong AI, we will likely not understand it initially.

Note: I'm personally not too worried about the AI apocalypse, but I think "we don't even understand neural nets" should cause more concern, not less.