I don't, and I think it's dangerous to assume that. Many open source projects have systems in place for publicly reviewing and approving code, but microsoft does not have a history of conducting public code reviews for their projects. I would never assume that publicly viewable code is being reviewed by others who know what they are doing if it's not obvious that they are.
> Now, anyone can go in and verify how it works. Any backdoor would be visible in the code
I would only trust those with a high degree of mathematical knowledge, specifically around the subject of cryptography to be able review the code in any meaningful way. The rest of us can verify that it builds successfully and, well, that's about it.
Note also that the theory of homomorphic encryption has been developed in the open scientific community; all schemes and security estimates are backed up by published papers, and multiple other publicly available implementations. SEAL itself has been publicly available since 2015, although under a non-commercial license. In 2017 Microsoft helped launch the HomomorphicEncryption.org consortium for standardizing homomorphic encryption (see http://HomomorphicEncryption.org). Today, this group consists of more than 300 scientists from around the world, including partners and participants from industry (Microsoft, IBM, Duality Technologies, Intel, Google, multiple start-ups), top researchers from universities (MIT, Seoul National University, EPFL, ...), and government (e.g. NIST).
AFAIK, looking at the code for a Dual_EC_DRBG implementation wouldn't look backdoor'd as the backdoor was in the mathematics of the crypto itself.
Even if the code itself had a backdoor, a carefully hidden one may go undetected. Like a missing "goto" or a single equals sign; `if x = y`. An example here: https://freedom-to-tinker.com/2013/10/09/the-linux-backdoor-...