Apple used 3D based face recognition that only works when your face is right in front of the iPhone. We are a ways away from distance 3D face recognition, and given the advances with deep learning and bigger data sets in 2D, it isn't even necessary.
Maybe it's changed, but I think that if you get a new device, it basically has to reindex all the faces iirc. Or maybe it's only stored in your own personal iCloud.