What sounds unsafe about having a locally bound port inside a container that only binds with an env variable getting set?
One can read every second week about cases where some backdoor that was meant to be used "only for debugging" landed in the end product and became a security problem.
Actually I usually suspect malice when something like that is found once again, as "who the hell could be so stupid to deliver a product with a glaring backdoor". But maybe there is something to Hanlon's razor… :-D