I remember bloom filters (http://en.wikipedia.org/wiki/Bloom_filter) been good for this sort of thing.
[]: http://ipowerinfinity.wordpress.com/2008/03/02/bloom-filters...
Seems a lot of these cool things came out in the 80s due to the limits of memory/computational speed.