Control of the list which is used to scan your messages has the same security properties as the code of signal itself and a threat actor that can exploit the system must control both that and access the match reporting system (signal servers). An actor that controls these does not need to abuse this system, it would be easier to just push a rat alongside signal.
> "You cannot create a back door that only the good guys can go through"
Perhaps for crypto protocols. Software systems do this all the time in form of software updates, usage monitoring and even unattended remote support accounts. And also, just because it might be possible to find a vuln in a system in the future that does not mean it is vulnerable at the time of design.
Not only are clean backdoors possible, software engineers can design them cleanly and make them sound nice and fluffy when they are the ones accessing the backdoor.