Realistically the security research community who find things like phone exploits are
very well accustomed to a lack of source code, and in many particular niches of infosec -- it's a baseline assumption for almost all analysis work. Apple's security team is the reason the iPhone is an extremely secure device, not because it's closed source, and not for lack of trying. Go ask people who find major security vulnerabilities -- binaries are rarely a true obstacle, time and mitigations are. (And you'll probably come away surprised at how easily most systems fall to even basic intrusive analysis, no source code needed.)
Source code is merely convenient for the security community in cases like this. But it's hardly a requirement at all, and you can absolutely build a secure system if you design for it.
All that said, I think source code for a device as personal as a phone is largely more of a question of ethics and this is what Purism heavily leans on as an angle for their marketing. I think that's a good angle, independent of any security aspects: owning the code and having the right to it (even if you don't ever touch it) is an important aspect of owning a device as deeply personal as a phone. People like us who believe that should totally advocate for it -- I think it's a fairly simple case to make. But it really has little to do with _security_ on its nose, if you ask me.
(Also, if your government mandates backdoors, they will not be thwarted so easily by simply recompiling AOSP or whatever and loading a new ROM, if they do not wish to be. Any belief to the contrary is a fairy tale that people like us enjoy telling themselves. But I'll give you a hint: writing a Makefile cannot get you out of jail, and you are not that good at covering your tracks.)