Because almost everyone prominent in the movement has had some combination of lack of judgment and morals. There various terms for rationally pursuing philanthropy; associating with effective altruism involves embracing its ideas, people and culture.
EA has many tenets, some of make sense per se, but collectively–particulalry when taking into account that EA isn't just an idea but a movement of people with a distinct culture and track record–probably do more harm than good.
Well, for one, the frequency of its members committing mass fraud.
Like, if a church preaches kindness and love but its preachers and members are constantly out there murdering people, it's a bit silly to point to the text alone when evaluating the organisation.
Like, if a church had 100,000 members, you would indeed expect them to commit a few murders each year. And GiveWell alone claims to have induced 150,000 people to donate... https://www.givewell.org/default/citations#Number_of_donors
Going to need a good source for that crowd having done more good than the frauds of even just their most-notorious members.
Masquerading as an altruist to justify massive management fees for running a fund of concentrated, leveraged positions is one of the oldest lipsticks on financial pigs.
And it does not come cheap. Someone who says "I will not work for Facebook" is taking a stand at the expense of significant personal gain.