Depends on the jurisdiction, but generally yes it'd be illegal.
You can compare it to opening someone's home because he left his key in a public place unbeknownst to him.
You generally couldn't make the argument that 'he may have wanted people to have the key, that's why he left it in a public/open source place'. Firstly, one can't assume that, so we must hear it explicitly before it's true. And secondly, if universal open-source access was provided, there wouldn't be a key to find as there wouldn't be a lock in the first place to allow specific access. The whole point of a password or key means you do not want fully open access to all, meaning any password or key in an open-source project is likely unintentional, a mistake, and thus you'd be entering without permission which is illegal in most jurisdictions.