This is a hefty bet. The gamble is that you're very good at this, and so you'll deliver better performance with HTTP/2 than the client could have got from HTTP/3, but even if you actually are right by a slim margin you're likely to suffer the same pushback from users as if you were wrong. The only scenarios where this is a smart bet are where competitors allow HTTP/3 and have noticeably worse performance.
If it's close, regardless of which way, you'll get beaten up for disabling HTTP/3, maybe that's unjust but that's how it is.