I must be missing something. You have 2MB of data for "is my 6 character SHA1 prefix in any breach". Why can't you distribute that to every edge nodes ?
Additionally, people don't just want a boolean answer of "was my email breached somewhere". They want a list of all the breaches that breached the email. So the returned data actually needs to be a list of emails and the list of breaches that each email was breached in.
>Via the public API. This endpoint also takes an email address as input and then returns all breaches it appears in.
The initial prefix check would probably reduce the amount of lookups necessary, as it would only be necessary to do a deeper search if the prefix matches.
There are 5 billion emails in at least 1 breach and 16 million prefixes. Almost all if not all prefixes have at least 1 email in a breach. So almost all prefixes match. I don't see why it's useful to spend a bunch of effort optimizing the very rare case of a prefix not matching.
Now, if the bloom filter checked emails instead of checking prefixes, that would be useful. However, a bloom filter of 5B elements with a 10% false positive rate would be 2.8 GB, which is prohibitively large.
Also regarding "you can get rid of a significant portion of requests at the edge with a bloom filter", Troy's existing design already gets rid of a significant portion of requests at the edge. That's why he says
>The response from each search was coming back so quickly that the user wasn’t sure if it was legitimately checking subsequent addresses they entered or if there was a glitch.
According to this calculator, a bloom filter for 16M elements with a 10% false positive rate would be 9MB.