You are equating "silent majority" and "hardcore right wing". That's not how it works in practice - this is something that both right and left can and do get wrong.
The "silent majority" exists in this case especially around wedge issues like trans sports inclusion [0].
[0] https://19thnews.org/2021/05/gallup-americans-oppose-transge...