Dealing with Duplicate Person Data (Levenshtein Distance, Nicknames & Multi-core/CPU)
proudtouseperl.com
proudtouseperl.com
Our Japanese university customers use the following to test identities for equality:
1) Same first name
2) Same last name
3) Same high school
4) Same birthday
5) Same high school major
6) Same high school GPA
7) Human review
Human review exists because if after all those checks we still merge two students and then someone fails the entrance examination as a result, someone at our company and someone at the university will pretty much have to resign in disgrace.
1. Hash the names (with something like Text::DoubleMetaphone) and select from this list rather than trolling the whole database.
2. Check out POE (great for distributed parallel processing)
3. Consider how you will handle null fields (a lack of information is not the same as badly matching information).
4. Consider the uniqueness of the data values themselves (Ebenezer Rumpelstilzchen is more unique than John Smith), but you have to watch out for foreign names (which seem unique but aren't).
5. Matching on multiple fields seems multiplicative, not additive.(HN really needs some CSS that restricts the width of code blocks.)
I tried with CSS alone, but as I recall it was inconsistent in multiple browsers, so went with js.
Regardless, it seems possible. Someone that actually has a clue about CSS and/or Javascript should submit a patch to fix HN :) (Unfortunately, I do not have said clue.)
Some other things we have used over the years to remove dups:
SOUNDEX (last name, first name, street name)
Zip Code (1st 2 digits, 1st 3 digits, all 5 digits)
Street Name only (discard numerics)
First letter of First name only (watch out, Joe and Jerry may be reduced to one record)
For mailing lists, are 2 people with the same last name at the same address really 2 people? (Sometimes yes, sometimes no)