I am also in the same category, but some games are made for the controller.

Admittedly, the only one I can think of at the moment is Eldenring: The PC (mouse+keyboard) experience was terrible.

I still passed on a controller but the controller first experience was glaring.