When I worked in the office before 2019 we had all kinds of people there. It was kind of beautiful in a way. Straight up communists discussing stuff with conservatives without any heat. Isolation breeds extremism.
It's true that it would close doors but man until the market takes that choice away from me I don't want to open those, and I say that as a guy who never worked with any of this even indirectly. And to be fair - not having experience with those things also closes doors, just different ones.
If you don’t take a job that would pay more you are also hurting your income opportunities. Turning a job down because of some hypothetical problem at some future date isn’t rational imo.
If you genuinely believe certain industries would hurt your future job prospects, you need to quantify the risk. And I have a hard time believing the particular industry would matter at all to an employer. If you can build CRUD apps for a gambling company you can build CRUD apps for a pediatric cancer charity.