This is almost certainly delusional, especially if it's a software patent. Although that may not prevent you from building a $100m business on it.
> and wants to maintain personal ownership of the patent.
This was one of the WeWork shenanigans with the trademark being held by the owner.
> I'm worried that if we start a company that uses the invention without owning the patent, our company could be killed at any time by the patent holder. While I trust my cofounder, investors may be turned off and the patent could always end up in the hands of a less benevolent owner.
These are reasonable worries.
The bare minimum for any company with multiple founders is to sort out the initial equity distribution. This may not be 50/50 if it's his patent, but it needs to be something you're both happy with. It's a difficult conversation, but it only gets more difficult if you're successful and there's real money involved.
The bare minimum for the company should be a permanent non-revokable license for the patent. He may want it to be non-sublicensable and non-transferable, which are reasonable requests.
If you can't satisfactorily resolve this between you, your working relationship is not good enough to run a business between you.
You also need to ask your co-founder: when someone comes and offers $10m equity investment into the business on condition that the company is fully assigned the patent, would they really say no?