According to the Post "Once you give it permission, an active app keeps on having access to your face until you delete it or dig into advanced settings. There’s no option that says, “Just for the next five minutes.”"
So a random developer can (kind of) look at us when we turn the light off. Not only that, those cameras work also in the sunlight. You can go on YouTube and search "clothes transparent to infrared". Not much, especially considering that the app is not getting a real image, but still somebody will be uncomfortable with that.
However there also legit and interesting applications for infrared cameras and they've been on sale for a long time. What's changing is the number of those cameras that are getting around.
[1] https://www.macrumors.com/2017/09/13/how-iphone-x-face-id-wo...