If Facebook is only using something like a hash of an email address in order to target ads at specific Facebook users at the request of one of their advertisers, they are probably only acting as a data processor for a very specific purpose that might be acceptable for both Facebook themselves and the advertiser under the GDPR rules.
If Facebook does anything else at all with that data, their role probably changes from a GDPR perspective. The hash is personal data, since by definition it's being used to identify a specific person. If Facebook is using the data they have associated with that hash -- for example, anything they know about the business that provided it -- to build up more of a profile on their users, they are probably now a data controller, possibly as well as a data processor in connection with the original targeted ad process. Then you get into questions about whether Facebook's users have given their suitably informed consent to Facebook or there is some other lawful basis for whatever processing is happening.
Obviously if businesses were providing actual email addresses to Facebook or if Facebook were using that data to do things like building shadow profiles on non-Facebook users, that would be another level entirely. And AFAIK, the custom audience tools on marketing platforms like Facebook typically do accept directly uploads of literal email addresses, phone numbers or other identifying details for the audience to be targeted, so maybe the discussion about hashing above is all moot anyway.
I think Facebook hashes any plain text you give it client side to (try) to buffer their defenses... but I’m not sure. It’s a theater.
I wrote a blog post about this data sharing with Facebook https://blog.gingerlime.com/2020/whos-sharing-my-data-and-wh... that digs a little in this direction.
https://www.lda.bayern.de/media/pm2017_07.pdf is the official press release of the Bavarian DPA (in German), with their guidelines attached (starting on page 4).
There is two kinds of "custom audiences" - one list-based and one based on tracking pixels. I'll only quote the parts relevant to the method where customer lists are uploaded.
a.Rechtmäßiger Einsatz - Der Einsatz ist nur aufgrund einer informierten Einwilligung der Kunden zulässig. Das Hochladen der Kundenliste kann weder auf eine Rechtsgrundlage des BDSG noch des TMG gestützt werden. Diese Rechtsauffassung beruht auf einer europarechtskonformen Auslegung der geltenden deutschen Datenschutzbestimmungen und berücksichtigt die jüngsten Entscheidungen des EuGH zum Datenschutz. Im Übrigen wird das Übermitteln dieser Liste an Facebook auch auf der Basis des ab Mai 2018 geltenden Rechts, d.h. nach der Datenschutz-Grundverordnung (DS-GVO), nicht ohne Einwilligung zulässig sein.
b.Widerruf der Einwilligung - Widerruft der Betroffene seine Einwilligung, so muss er von der Kundenliste entfernt werden. Da der Webseiten-Betreiber keine Kenntnis davon hat, welche Kunden auch Nutzer auf Facebook sind und beworben werden, ist die vollständige Custom Audience-Liste unverzüglich zu aktualisieren.
(Translation - Google translate with misleading issues corrected manually:
Lawful use - Use is only permitted with the informed consent of the customer. The uploading of the customer list can neither be based on a legal basis of the BDSG nor the TMG. This legal opinion is based on an interpretation of the applicable German data protection regulations in accordance with European law and takes into account the most recent decisions of the ECJ on data protection. Beyond that, transmitting this list to Facebook will also not be permitted without consent according to the law applicable from May 2018, i.e. according to the General Data Protection Regulation (GDPR).
Withdrawal of consent - If the person concerned withdraws his or her consent, he or she must be removed from the customer list. Since the website operator has no knowledge of which customers are also users on Facebook and are being advertised, the complete Custom Audience list must be updated immediately.)
You're also telling Facebook "by the way, I have a relationship with someone with this email address". That's personally identifiable information that you're sending to Facebook. Under the GDPR you can only do that if you have the explicit and freely given opt-in permission to do that from each respective person. "By using this site you agree to..." or "by signing up you agree to..." does not qualify as consent under the GDPR.
If the person does not live in Europe and you are not in Europe then the GDPR doesn't apply, of course.
If I'm not on Facebook (which I'm not) you are telling them that there most likely exist a user with this email address and an interest in your service. If many companies do this FB might even be able to build a profile of me without me doing anything
This is (or at least should be) not Bueno under GDPR / data minimalization.