There are several senses to "politics". You are right in that, in the general sense, politics is any collective negotiation of what matters to the group. There are objectively optimal ways to achieve a goal, but choosing what goals to pursue, and what benefits are worth what costs, and who they will affect, that is rather subjective and the realm of politics. In that sense politics is fundamental and valuable.

But in a more concrete sense, politics also refers to our tendency to join opposing teams or tribes and fight it out, more or less literally. In that sense, "everything is political" can mean viewing everything as a fight between groups, or worse, associating everything to the conflict between the two dominant groups. That is quite toxic.