It has ZERO impact on my judgement of the content, discussion or presentation. I do not believe that a person's gender provides any meaningful impact to a presentation unless the topic is specifically about that person's gender.
To think otherwise is sexist and discriminatory.
Now if the best participants for panels are female and are being actively excluded because of their gender, this is a problem. But simply refusing to go to "all male panels" is a useless call to action and will not solve anything. Neither will forcing a gender count.
Find specific instances of qualified and capable women being overlooked by organizers in favor of less qualified men simply because of gender stereotyping and then publicize the crap out those incidents. Boycott those specific conferences.