Yes, this is really hard.

You could get a Solarwinds type situation where the adversary has the signing keys and ability to publish to the website.

You might also find that the vendor ships a library (like libxz) as a part of their invisible or hidden supply chain, that is able to be compromised.

You might find that one of the people working at the company makes a change to the code to enable remote access by the adversary in a targeted collaboration/attack.

The problem isn't that signing key (although I could delve into the lengths you'd need to go to to keep that secret under these threat models) - the problem is what they sign. A signed end release binary or series of packages isn't going to address the software source code itself having something added, or the dependencies of it being compromised.