You have to trust someone, at some point. HTTPS/SSL relies on a distributed trust system. Even with crypto, you have to trust the developers that created the protocol and the consortium of groups that operate it.
Actually, no. Plenty of SSL/HTTPS servers run non-open sourced software. Plenty of open source foundations have closed decision making processes. Being open source is neither necessary nor sufficient to being trustworthy.