I'd actually be fine with criminals using this to leave my country...

Housing and feeding criminals is expensive. Letting them leave permanently is in my eyes a better solution.

Which would become an one-way route, because once a nation has become a criminal-refugee paradise, why would third countries accept them travelling to their shores?