This exactly. To have a more secure device this is one of the trade offs. Can you imagine the articles if you could swap out faceID systems to unlock an encrypted iphone?
[1] https://www.apple.com/business-docs/FaceID_Security_Guide.pd...
Yes, the articles would go something like this: "WTF is wrong with Apple, did they intentionally implement 'security' in the worst possible way, by leaving the phone unencrypted and just using faceID as a lock screen?!"
That is what they'd have to do for your statement to make any sense, they'd have to leave the data unencrypted and just use a removable component as a pass or fail doorman. So the system would have always been unsecure, it would just be more obvious in this scenario.