Equally, one might say that you don't know how the world works and how these scores will be governed by the few to get power over the many.
You can also buy whatever stocks and shares you want. ESG does not prevent companies from listing.
If you don't want to invest based on ESG then don't.
If these concerns people have about ESG are so important, they should be passed at the democratic level via legislation. At least that way, people could debate and change the system as needed. You didn't need ESG to get rid of lead plumbing - just an interested government.
Of course these days, we can only have private power, not public power.