The Correct Way to Validate Email Addresses
hackernoon.com
hackernoon.com
Surprisingly, the validation is often done 100% client-side anyway, and simply modifying the incorrect regex lets my email address through... If I wrecked havoc on your back-end, then it's your fault for sucking ;)
Scarier still is when it's a server-side response that rejects my password for its contents...
Why: Because if they were (at least) hashing it the output from the hash would be a binary string in which case they would have to be 8-bit clean through to the DB column where the hash output resided, and then there would be no reason to care what character was in the input.
Caring specifically about a + in a password also implies that their authentication might be setup internally as a http endpoint with URL encoding of a "password=" form variable (because + is used as the escape for hex encoded characters in URL encoding).
Both, of course, imply a lack of proper secure design in their password handling.
I don't think that is true at all.
I may very well want to put a few simple rules I validate serverside, such as
1) No username in password 2) No email in password 3) No list of 100 most common passwords in password
All of which require me to look at the text for your password, none of which mean I am storing it in plaintext.
I've run into a couple of sites that reject passwords containing ', %, and other special characters that suggest there may be some truth to it.
If you're scrubbing input as if it's about to be insert into a database via SQL, then there's really only two possibilities. Either a) you're running legacy code that still does the check and does blind escaping (which has its own set of implications) or b) you really truly are storing passwords as plain text.
There is no reason to assume rejecting of a + means they don't hash, and browsers escape the password for you in POST/GET etc. The fact that + is used for space really isn't relevant.
(That said, I think it's pretty stupid to "validate" passwords beyond checking for "12345", "password", a min length, and other public info. e.g. your password should not be your name or date of birth. Anything any password generator spits out, so long as it meets the min length criteria, should be accepted).
Speaking of annoying validations, my name has a hyphen in it but you'd be surprised how often that's rejected with the familiar, "Please enter a valid last name." Sigh.
Btw, almost nobody is actually storing hashes as binary blobs. Pretty much every framework I know stores either base64, or direct hex encoding of the value.
Why? Assuming that you have established a secure connection with that server (i.e., HTTPS by means of TLS), then it is perfectly fine for the server to check, at the time you are setting the password, if your password confirms to the rules established. And when the password turns out to be suitable, it is okay for the password to be sent to the server as-is.
Now it goes without saying that as soon as you have picked a suitable new password, that the server will store only a (proper) hash — by using BCrypt for example. At no time is the plain text password stored anywhere, and any proper HTTP API will send the password via POST to prevent it from being logged in the server's access logs (which is where it could end up if sent as parameter via a GET request).
There are plenty of services that screw up and try to apply nonsensical rules (such as limiting the length of the password to anything less than, say, 256 characters), but in general this is done to prevent weak passwords. You can't exclusively do this client-side, because in security terms, the client cannot be trusted to actually apply the validation and to generate a proper hash; the client can be bypassed. Of course the client can and should validate any input before the server gets to it, so ideally the server need not even come into it during validation, but the server has the final word.
As user though, you have the responsibility of not reusing passwords anywhere (and you don't, because you sensibly use a password generator as you mention). Don't assume that any service handles security well.
Hopefully. When I see rules limiting passwords to 16 characters and disallowing SQL special characters, I'm having doubts.
A friend's project decided to disallow umlauts, combined characters like ´e (can't type the correct e with accent mark), the pipe symbol and a couple more in new passwords.
Not due to plaintext storage or so, but because of customer service issues - people were bugging support all the time because they were e.g. abroad and couldn't enter umlauts or on a Mac and couldn't find the pipe symbol (it isn't written on the keyboard!)... of course, a quick direction to the Zeichentabelle in Windows helped, but what for Mac users? At least it did take a huge load off the customer service guys.
I'm looking forward to the date when some clueless user will input a UTF8 emoji as a password, given that Android and iOS keyboards now include these on special keyboards...
Looking forward? That happened a year ago and caused problems because the OS X Yosemite login screen had no way to input emoji:
http://apple.stackexchange.com/questions/202143/i-included-e...
You mean é ? :-)
By pipe, you mean "|", right?
That could be confusing to someone who doesn't know better.
Now, why would you preemptively (and explicitly) throw out a candidate password string based on its characters?
At least make the validation consistent.
I also had trouble with connecting to a wifi network with a WPA password containing an ampersand in Arch Linux using the Arch Wiki's recommended command line network manager utility (netctl, if my memory serves me correctly), and I tried all sorts of ways of escaping it and quoting it in the configuration file.
The Arch devs on IRC just told me to change my AP settings so that the wifi network doesn't use a password containing an ampersand character. Well, that would have been a good idea, but it wasn't my network, so I didn't have the ability to change its WPA password.
Which you shouldn't. execve (and friends) will let you pass an explicit array of flags, and you should always use one of those functions if you're calling another program. No interpretation.
(There are some characters that are treated specially by shells, and require escaping - but you don't normally spawn child processes via a shell.)
Why: Because if they were (at least) hashing the password, then the output of the hash would be a fixed size token unrelated to the length of the input plaintext password, and no such arbitrary short limit would be necessary on the plaintext password itself.
It's as though the developer only ever thought of how people wouldn't meet their minimum requirements, wrapped it all in a case or switch and let the default be whatever they hadn't checked in previous conditions.... not thinking that anyone would exceed their requirements.
Very annoying and sloppy.
Bets on how long before those folks make the news for losing a hundred million unhashed passwords?
I can't think of a good way for a business that has an online interface and frequently handles phone calls from customers to validate that they're talking to the correct person. Asking for other personal information can be used by an attacker to compromise multiple accounts via social engineering: http://www.wired.com/2012/08/apple-amazon-mat-honan-hacking/
The best guess I've seen is that they might be hashing, but also having people read passwords to customer service reps. That would justify caring about rude plaintext, but it's also a terrible system.
Terrible security, sure, but at least it came from the desire for usability, not just basic encryption idiocy.
The text of the password should not persist.
I tried to explain to my boss why this was such a terrible idea and he was not hearing any of it.
In disbelief I logged out, tried again but this time I intentionally didn't use any special characters. It worked....
Account synchronization was involved - IIRC the initial address confirmation message got thorough.
The problem here, though, is that the email sender just stripped the + and sent the content to foobar@gmail.com. That's just a totally different account, chosen without warning.
A lot of people either had hardcoded lists, or they checked the length and refused TLDs longer than 3 characters. We kept e-mailing people about it, and kept getting messages back from people who had "fixed" it by adding just us (we'd generally have pointed people to articles listing other new TLDs too, to make it clear to them that this wasn't just one TLD), or increasing the limit to 4 characters... We quickly gave up on trying to get these people to stop having useless checks and settled for just getting them to accept ours.
It's quite shocking people still haven't learnt even after the number of expansions since.
\.[A-Za-z]{2,4}$ (this is the end of the regex in Angular 1.15, it also doesn't support brackets or quotes in the local part, poor showing from Google)
At the time when they were written, gTLDs didn't exist, so the longest a tld could be was 4 characters (.info, .mobi). It's pretty piss poor futureproofing to be honest.
But it does get awkward quite a few times when having to interact with a human (customer services, hotel bookings, etc) via email or phone when they get a bit confused why their company name is my email alias....
Spelling a long alias over the phone letter by letter is especially tedious...
Also responding to emails either means I have to configure yet another sender alias, or mostly just send from my normal alias, which sometimes gets rejected or confuse whomever I interact with.
Also hate unsubscribe links that insist on sending unsubscribe email from that alias(mailman etc).
I had that problem so many times that I whipped up a quick Rails app that generates new email addresses. Type in the company name, hit submit. It uses a random project name generator gem to create something like SteelyFishSauce@<mydomain>.com, displays it on the screen, and emails "<Company name> has been associated with SteelyFishSauce@<mydomain>.com" to the spamcatcher address.
The nasty part is replying with the main address as the sender. I'd really love to have email clients with reasonable support for this usage pattern for the platforms I use, but maybe then the pattern might become popular enough to lose some of its advantages.
Originally, I wanted to create a Postfix filter based on an HMAC together with a browser extension which would simply let me generate new valid email addresses in the form of prefix.HMAC(secret, prefix)@example.com but I have never implemented it.
But I have an acquaintance who is constantly bickering and venting about this issue. If he strongly suspects that a signup form will reject his plus address, he will enter it even more fervently to prove they are idiots.
I'm firmly on the side of "Too bad. Maybe you could simply move on with your life? Just use a dash if you need sub-mailboxes"
Why are you so firm on this position when it seems to be a legitimate problem for him?
What is easier and better for your nerves? Moving to another mail provider or getting worked up for years?
Or if moving is not an option: how about stop wasting time and nerves on it? Just accept that your mail address doesn't work everywhere. There is no possible uture where you will force everyone on earth to stop mis-validating email addresses!
And I've never disputed that the problem is legitimate. I actually said in my comment!
Speaking as someone who uses + addresses to filter stuff from mostly well-known websites, I have never seen this. I have seen this a few times on old, crusty, finance websites etc. but I hardly ever need to use a + address with them anyway. (It does make me wonder about how good their internal security is, though.)
It's less common, but some websites ask you to type your email address once they realise they can't find you on record. If the "+" doesn't work in that field (server side rejected), then you can try %2B there as well.
Also, there's always https://mailhero.io/
Interestingly, though, the support email address they had hanging around at the bottom of the login page actually routed to an engineer (somewhere), which was nice, although they did claim repeatedly that the "+" character wasn't valid in an email address, which is of course not true. I was floored just to get a response from my complaint, though, even though it didn't seem likely to lead to them fixing the problem.
I find common culprits are service companies; gas, electricity, real estate websites, insurance companies etc.
EDIT: Oh, and warranties! Man warranty websites tend to suck. Samsung for instance won't let you register your product to an email with a +.
In 99% of those cases, it prevents someone from entering a wrong email.
We do not do email activation by forcing people to click a link in their email to validate that they received it since that causes a drop in the funnel and reduces the amount of revenues (non technical users tend to not come back when you ask them to go to their emails to verify it). So, in this case, correcting the typical typoes is very important. In our case though the information is not extremely private so it's less of a problem to do this.
Having people type the email twice doesn't really prevent typoes, people copy and paste. And if you disable paste, then it becomes annoying to users and you don't want to annoy users during the signup process (plus I hate websites that mess up with paste so I won't be hypocritical and do it).
Lastly, I know that having an email with a local domain name with no TLD is valid but it'll never be valid in the context we are sending so supporting them just doesn't make sense.
That can be helpful, but I assume you suggest the user that they may have made a typo and not flat out reject the input?
> […] having an email with a local domain name with no TLD is valid but it'll never be valid in the context […]
Well, depending on how the fairly recent brand generic top-level domains work out, I would not rule out someone actually using ceo@cocacola at some point (until the CEO of Coca-Cola realises that legacy email validation regexes mean a suspiciously quieter then usual inbox).
> I would not rule out someone actually using ceo@cocacola at some point
Of course, this is subject to change if the practice changes. The aim is to make sure that we balance the percentage of emails that are correct versus the number of potential rejections. If 0.00001% of users use a top level domain and 99.9999% of emails without a tld are invalid, then it makes sense to reject it. We're not trying to find the most perfect solution just the most convenient solution for users.
But if you include plain text password in it... ugh.
But I can see how there may be other concerns (commercial or not) that interfere with this.
Honestly, it's ruthlessly pragmatic business reasons. Is it ideal? No but in the end, it's the most painless for most users...
And, also most users on that particular site have little computer experience, so it does affect how they react and how we work. If it were something targeting the HN crowd, I'd probably have enforced email validation and would see a much lower drop in conversion due to it because people on HN are used to that and do not have a problem with it.
So, validations and UI workflow have to be adapted to your audience and your business and that's the main point really.
In this particular case, we did experiments with enforcing email validation for a subset of new signups or even allowing a small number of signups to signup with a username instead of an email. So we do have data...
Most of the work I do for other customers doesn't touch those areas of their app, so I only have relevant on hand experience on that particular site but I think it's important to mitigate the recommendation of "Always validation emails" or "Always use regexp" and try to think of the best experience for the kind of users you're targeting.
Sometimes in HN, people talk in absolutes when things are instead very context-dependent.
Have you considered doing email validation without forcing users to comply? It's fairly common to require valid emails after X days, or in order to unlock N features. Gets the advantages of having valid emails without the disadvantages of a drop signups.
In other words, it does make sense to check that they entered an '@' symbol somewhere, since it shows that they understood it was an email field. Any 'validation' beyond that is useless.
@ signs are also permitted in comments: (ted@home)ted@example.invalid is also a valid email address.
I got a popular givenname.familyname@gmail.com address and I frequently get mail that's meant for other people who share my name. The vast majority of the time it's the individual themselves who sign up for a service or offering but there's rarely a validation upfront.
The best emails are the ones who extend full trust to the email recipient over some account during that first email. Facebook, shame on you.
Well what else would you have them do? Have people enter their address and send letters there to do a password reset?
A confirmation email before full trust is going to do little: a malicious person would just click that link, right?
There are an amazing number of C. R. Conrad's in the world -- mostly the USA -- who seem to think they own that address.
For being able to do password resets later, permission to add to mailing list, avoiding sending private info to the wrong user, avoiding allowing someone to masquerading or impersonate someone they're not.. All should be validated.
If you're looking for a username as login identity and nothing more (and you don't have password reset functionality), then ask for a 'username', not email addresses.
Mailing list - reluctant OK
"avoiding sending private info to the wrong user" - how could this happen?
"avoiding allowing someone to masquerading or impersonate someone they're not" - how could this happen? Like if I signed up as tim@apple.com? What could realistically happen?
Failure to do so means:
- The user who actually owns that e-mail can't login
- The user who signs up can potentially impersonate the real user (depending on what your app does)
- The user who signs up can't reset their password
- The user who actually owns the e-mail can take ownership of the account (by resetting password)
Not true. There are people who share an email address.
And when they do, it's usually mine.
With no validation other than sending the email, you have to know, for example, what the server would do with an email address that claims to be @localhost. Now it becomes a problem- or at least a question and concern- for the backend system. Whether the backend interprets root@localhost as valid and does exactly what it's told or rejects it due to some configuration- it has become a backend complication and a DOS attack vector.
A simple policy of only handling a subset- the common class of email addresses- is one of the things that allows us to have a simple mental model of what the MTA is supposed to do. The fact that it sometimes caught a type-o, or not, is incidental. "Invalid email" wasn't meant to imply the email address doesn't fit the spec- it was meant to imply that a particular site or service has chosen not to accept email addresses like that.
Or at least that's what I assumed :-)
Sanitization is at best idiotic, at worst creates security problems. There is no such thing as "bad characters", there only is broken code that incorrectly encodes stuff. If you ever find yourself modifying user input "for security reasons" (or really, for any reason at all), you are doing it wrong. The only sane thing to do is to make sure that the semantics of every single character of your user's input is preserved in whatever data format you need to represent it in.
2. Rejecting @localhost addresses doesn't really make a whole lot of sense. People could just enter the public IP address or hostname of the server, or add a DNS A record under their own domain that points to 127.0.0.1, or an MX record that points to localhost, or any number of other weird stuff that you could not possibly validate anyway (if only because it could be changed at any point lateron). Just configure your mail server properly and then send the damn email, and if it does get sent to root@localhost, and possibly forwarded to the admin--so what? People obviously could just sign up using your admin's email address anyway, and that not only at your site, but at millions of sites out there, you won't be able to stop them. There is nothing particularly dangerous about receiving unsolicited signup emails or about sending emails to yourself.
Depends on what you do with them, in the latter case. There could be an amplification attack there.
Validating domain parts to a certain extent isn't a bad idea, at least as far as non-routable domain names and RFC1918 ranges go. I've seen this done (actually implemented some of it, in fact) at a past employer, who were basically looking to cover the 90% case in terms of not getting hosed by a trivial attack. It doesn't take much effort and it makes 4chan's life harder. What's not to like?
The important thing is to not change information. "Sanitization" as it is commonly used means doing something that (potentially) changes information. Which is in contrast to decoding/encoding/parsing/unparsing/translation/..., which, if done correctly, change representation, but not information.
So, to make it a useful distinction, I would call anything that potentially changes the semantics of the processed data "sanitization", and avoid using the term for anything else.
So, simply parsing a string with an HTML parser, possibly checking for acceptable elements, and then serializing back into some sort of canonical form that is semantically equivalent to the input, that's perfectly fine, and I wouldn't call that sanitization, but rather validation and canonicalization.
If you simply start dropping elements, though, that's probably a bad idea, just as simply dropping "<" characters is a bad idea, because those elements presumably bear some semantic meaning, just as a "<" in a message presumably bears some semantic meaning.
Now, it is not always obvious which level of abstraction to evaluate the semantics (and thus the preservation of semantics) at. So, it might be prefectly fine, for example, to remove or replace some elements where the semantics are known and you can show that, say, removing emphasis still generally preserves the meaning of a text.
But a whitelist approach where you simply remove everything that isn't on the whitelist usually is a bad idea. If you want to have a whitelist, use it for validation, and reject anything that's not acceptable, so the user can transform their input in such a way as to avoid any constructs you don't want, while still retaining the meaning of what they are trying to say.
Edit: here's a good example. I don't know if they still do this, but when I worked at Yahoo!, they used a modified version of PHP that applied a comprehensive sanitization process to all user inputs. As a frontend coder at Y!, all the information you pulled from request parameters, headers, etc, ran through this validation at the PHP level before your app code got to it. You can then literally splat this information into an html page raw, without any further treatment, and not expose the Y! property you work for to an XSS or other injection vectors. There were ways to obtain the raw input using explicit accessors when needed, and these workarounds were detectable by code monitoring tools and had to be reviewed and approved by security team(s). Overall this worked really quite well, in my opinion. Y! could hire junior frontend devs without deep knowledge of data encoding, security issues, etc etc and rest easy. I think the principle of safe-by-default, even if it means destruction of user input in some cases via aggressive sanitization, is a good principle to apply to a frontend framework.
However, IETF made lots of progress simplifying things- to the point where, at the very least, the standard tells us specifically that we should leave it up to the destination host to interpret the local part of the email address-- that is, the thing to the right of the @ should be given the thing on the left unmolested ideally- even being ignored by intermediate relay servers. Since that's what most people complain about, any validation to the left of the @ should become extinct.
But off the top of my head that still leaves the thing on the right of the @ (such as localhost), buffer overflows by allowing longer strings than the standard allows (those limits do exist), and the problem with multiple @ which the MTA may or may not handle well... Since I'm not a security expert I'm going to go out on a limb and assume that I'm missing a bunch of other things.
My point is not that the article is wrong, though- my point is that if he wanted to convince me to only validate by sending the email on any string, he should convince me that those security concerns are not an issue- not that it's not good at catching type-os.
The currently operative RFC is 5322, specifically section 3.4.1: https://tools.ietf.org/html/rfc5322#section-3.4.1
There are some basic rules that you could safely apply to an address, which would prevent some attacks (buffer overflows, etc.) while also not blocking any legitimate addresses. E.g. limiting the overall length to 255 characters, for instance, could be defensible practice. There are also well-defined rules for validating the domain portion, since it has to be a routable address by definition.
What nobody ought to be doing is looking too hard at the string to the left of the @ symbol, because it's designed purely as instructions to the recipient server. Nobody else needs to care about it; only the receiving mailserver needs to actually parse that part of the address, in order to put the message into the right mailbox. From what I've seen, the vast majority of false-positive validation failures occur because people are looking at the mailbox portion of an email address when they have no business doing so.
Email addresses aren't a special enough case to be handled differently than any other user input, which we always validate to both sanitize and show client-side errors, if nothing else.
Sure, the complete regex is complex, but it is defined and is hardly unconquerable. Look at Django's `EmailValidator` implementation for example [0] that is mature and well tested [1].
The author has not convinced me that ignoring validation is the right choice when options with a scope so thorough exist.
[0]: https://github.com/django/django/blob/master/django/core/val...
[1]: https://github.com/django/django/blob/a9215b7c36bff232bcc941...
If it is a regular expression, then it is not able to match all valid email addresses, because the grammar of email addresses is context-free, and regular expressions can only match regular grammars. It doesn't matter if it is defined or not: if it's a true regular expression, then it simply cannot validate email addresses.
(it may, of course, be a context-free expression masquerading as a regular expression)
I wonder if the django validator will choke on perfectly valid email addresses such as (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)
I suspect that it will, but of course I could be wrong.
I don't think you're really correct about "email addresses" being context-free, or at least, citation, please?
When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`. I don't think most people are going to think they can enter that, nor would most programmers even think to implement it. And I certainly wouldn't want to try explaining it to a PM…
If you accept that assumption, what about `addr-spec` isn't regular?
Also, using that assumption, your "perfectly valud email addresses such as …" would appear to not be valid, as it has unbalanced quotes. (In fact, even under the grammar of `address`, I'm not sure it's valid; it feels like it should be invalid for the same reason, but I've not rigorously checked this.)
> When I look at a generic "email address" entry field on a random form on the Internet, say on the sign-up page for some hot new startup's service, I expect it to take what RFC 5322 §3.4.1[1] calls an `addr-spec`; specifically, I don't ever expect such fields to take the grammar of what that RFC calls an `address`.
Well, sure. Let's look at what RFC 5322 defines as an addr-spec[1]:
addr-spec = local-part "@" domain
local-part = dot-atom / quoted-string / obs-local-part
dot-atom = [CFWS] dot-atom-text [CFWS]
CFWS = (1*([FWS] comment) [FWS]) / FWS
comment = "(" *([FWS] ccontent) [FWS] ")"
ccontent = ctext / quoted-pair / comment
Pair-balancing, of course, is impossible with regular expressions, since matching pairs requires push-down automata (which match CFGs) and cannot be done with finite-state machines, (which match regular expressions).
QED.
> Also, using that assumption, your "perfectly valud[sic] email addresses such as …" would appear to not be valid, as it has unbalanced quotes.
Nope, there are no unbalanced quotes in (this)"()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest): the first quote balances with the third, while the second quote is one of a quoted pair \" (which is allowed within a quoted-string, which is allowed within a local-part). It's all allowed per the spec.
I'll admit that it's a bit surprising, but it's true. One simply cannot match a valid RFC5322 addr-spec with a regular expression. One can, of course, match it with something which pretends to be regular but isn't really (as I noted).
Well done.
You can find multiple short email validation snippets using Lua LPeg pretty easily, but this is from the Lua creator's talk about LPeg which includes a part about proper RFC822 validation and how complicated it is for regex, but can be concisely done with PEGs.
http://program-transformation.org/pub/WGLD/Austin2012/Robert...
Here is the entire PEG slide. (Looks CFG-ish to me.)
address <- mailbox / group
group <- phrase ":" mailboxes? ";"
phrase <- word ("," word?)*
mailboxes <- mailbox ("," mailbox?)*
mailbox <- addr_spec / phrase route_addr
route_addr <- "<" route? addr_spec ">"
route <- ("@" domain) ("," ("@" domain)?)* ":"
addr_spec <- local_part "@" domain
local_part <- word ("." word)*
domain <- sub_domain ("." sub_domain)*
sub_domain <- domain_ref / domain_literal
domain_ref <- atom
domain_literal <- "[" ([^][] / "\" .)* "]"
word <- atom / quoted_string
atom <- [^] %c()<>@,;:\".[]+
quoted_string <- '"' ([^"\%nl] / "\" .)* '"'Are those parens really permitted in the host name? I'm looking at RFC 3696.
> 3. Restrictions on email addresses
> .... The syntax of the domain part corresponds to that in the previous section.
and
> 2. Restrictions on domain (DNS) names
> Any characters, or combination of bits (as octets), are permitted in DNS names. However, there is a preferred form that is required by most applications. This preferred form has been the only one permitted in the names of top-level domains, or TLDs. .... The LDH rule, as updated, provides that the labels (words or strings separated by periods) that make up a domain name must consist of only the ASCII [ASCII] alphabetic and numeric characters, plus the hyphen. No other symbols or punctuation characters are permitted, nor is blank space.
> Are those parens really permitted in the host name? I'm looking at RFC 3696.
RFC5322 states that an addr-spec consists of a local-part, followed by @, followed by a domain. It states that a domain may be a domain-literal; a domain-literal may begin and end with commented folding whitespace (CFWS). It's all in https://tools.ietf.org/html/rfc5322#section-3.4.1
It's only the dtext portion of the domain-literal which must consist of printable ASCII characters, not including [, ] or \.
Yes, foo@bar!baz!quux is a valid RFC5322 email address (decimal 33, i.e. !) is allowed, per the dtext production of the spec) — and so is foo@(comment (nested comment (escaped \(comment)))bar!baz!quux(another (comment (to prove) the point))
Whether one should ever actually use such an address is, of course, another matter entirely.
The possibility of having a valid email address where the second-level domain component can't actually be registered as a domain, to the best of my understanding, eg "bar!.com", is interesting.
If it doesn't, remind me to patch it to not accept that.
I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
You'd be wrong to do so. The whole point of RFCs and Standards is to take things out of the realm of personal preference.
Also, I suspect that a validator which allowed reasonable addresses like jim(somesite)@foo.invalid (which is both a good use of comments and what the + hack emulates) or "Ted Smith"@bar.invalid or "work@home"@jobs.invalid or "William \"Bill\" Jones"@baz.invalid but disallowed unreasonable ones would also be able to solve the halting problem. How many perfectly reasonable allowed features must one use before it's considered abuse?
> I have strong views on email validation, and they include telling people who use that kind of address to go register on someone else's site.
An email address is, simply, an address which can be given to an Internet mail server in order for it to properly route an email. All of the addresses I gave are perfectly valid ways to do so. An addresses is a property of the addressee, not of the addressor.
In other words: it may be your site, but it's my address.
Many of the more arcane things that can technically be done in an email address seem to me to be in the "turned out to be bad ideas" bin, and I have no problem making them be de facto deprecated even if no RFC has yet caught up to that.
Also, no halting-problem issues at all; my preferred approach would be to disallow certain classes of characters.
Technically-correct is the best kind of correct:-)
But I do think it's important to note, since there really are differences between regular and 'regular' expressions.
I wish that were the case. http://www.regular-expressions.info/refunicode.html
(They used to have a much more useful and concise comparison table but I can't for the life of me find it.)
I guess what I really meant is that programmers are talking about PCRE in terms of power, not in terms of exact syntax. In particular, they have recursive patterns, which are sufficient to pull them up to context-free grammars.
>>> from django.core.validators import EmailValidator
>>> EmailValidator()(""""()<>[]:,;@\\\"!#$%&'-/=?^_`{}| ~.a"(is)@(valid)example.org(honest)""")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/core/validators.py", line 203, in __call__
raise ValidationError(self.message, code=self.code)
django.core.exceptions.ValidationErrorSeriously, though, this is why regexp-based validation is Not Even Wrong. It's like showing up to a gunfight with a keen eye for detail.
2. Remove everything between + and @
3. Do validation
I'd never looked closely at Django's email validation, but after looking at it I'm inclined to stop using it. Example reason:
> # max length for domain name labels is 63 characters per RFC 1034
There is a high likelihood that most of your users have shorter domain names, and also a high likelihood that most if not all registrars currently enforce this. But it also used to be highly likely that domains didn't begin with a numeral, and even though it was disallowed, 3m.com existed.
Another: neither the tests nor the patterns account for unicode, which is valid for domain names irrespective of the local part, and if not allowed for the local part by existing providers almost certainly will be eventually.
Also, a thing I noticed in the Django tests that is just upsetting:
- name: `TEST_DATA`
- values: `(validator, value, expected)` (a validator of the data is not the data)
- unstructured groups of such that should almost certainly be structured
Without protocol changes, DNS cannot support labels longer than 63 bytes.
However, I agree that validating it is unnecessary.
Back in the 90s, we ran the customer rewards program mailing list for a mainstream business you've heard of. A [gnarly] regex took care of the gross failures but we still had double-digit percentage of invalid addresses and many spam reports because people mistyped their username, used their old ISP's email address which had been disconnected, etc.
The only approach which produced satisfactory results back then was to have a bit of (horrible) PHP code which did a full SMTP connection to deliver the welcome email. Even that wasn't enough to ensure you'd delivered it to the right person, however, so we had to track clicks on an activation link.
I would be surprised if the average internet user has gotten significantly more reliable over the last couple of decades and that's born out by the wide variety of misdirected legitimate email my shortname@gmail.com account receives.
The other key step is having some contact form which doesn't require the same test so people can report bugs if you get the logic wrong.
2. Getting billed for sending emails? WTF?
An email address which doesn't validate against the applicable RFC is ipso facto obviously invalid.
Validating against the RFC is somewhat complex though.
Also whether you use RegExp on email or not totally depends on what you're doing with the address. Are you throwing that email into Salesforce? Great! Salesforce does RegExp validation on all email addresses and the field is required. You _HAVE_ to do it. Solution? Use the same validation that Salesforce does. That's what I do.
Also the OPs math about how common invalid email addresses are is wrong. Do you know what the most common way of screwing up an email address is? Hitting tilde when you're tabbing away from the field. And that error is in the tenths or hundreds of a percent range (which is still significant). I have a dataset of 5MM email addresses typed in by jaded call center agents to prove it.
Yes, they aren't special, and you shouldn't validate any of the other stuff either, unless you actually need to understand the semantics. So much breakage happens because people think that they need to validate all kinds of stuff, it's not funny. People have their legal real names rejected because they're "not a valid name", people have their correct postal address rejected because it's "not a valid postal address" ... just forget it, if someone tells you their name, just believe them it's their name, if someone tells you their postal address, just believe them it's their postal address, if someone tells you their email address ...
Oh, and don't ever think about "sanitizing" stuff. Just don't. If you think you need it, you are doing something wrong. The solution to SQL injection is not to disallow people using "'" characters in messages, the solution to cross site scripting is not to prevent people from using "<" characters in comments, ... that's what encoding/escaping is for.
I ran a fresh install of Django to confirm and it behaves as you suggest.
Edit: Perhaps only for low digits. Hmm.
>>> from django.core.validators import EmailValidator
>>> e = EmailValidator()
>>> e.__call__('foo@[1.2.3.4]')
>>> e.__call__('foo@1.2.3.4')
ValidationError: [u'Enter a valid email address.']
>>> e.__call__('foo@0.0.0.0')
ValidationError: [u'Enter a valid email address.']
>>> e.__call__('foo@[0.0.0.0]')
>>> e.__call__('foo@10.10.10.10')
>>> e.__call__('foo@[10.10.10.10]')
>>> e.__call__('foo@255.255.255.255')
>>> e.__call__('foo@[255.255.255.255]')* somename@gmail.co
* anothername@yhoo.com
* myemail@hotmial.com
These are very common errors that occur nearly every day. A regex isn't going to help here. What does help, is a notification that asks people to verify what they typed –– if the email contains an obvious, common error, such as one listed above.
Just goes to show that assumptions are often wrong, and you have to crack open the spec document from time to time... [1]
Then there's the textbook company that lets me register but refuses to let me reset my password claiming that I'm trying to enter an "invalid email address."
We were unable to finish validating your connection with the "*.net" email server.
Yet I've been signed up for the pass holder announcements and get the email on a regular basis.
I gave up on it, not because of computer validation but because of stupid people! Nobody would "get" the .to domain and they'd always think there should be a .com or .gmail.com on the end of it. Especially older people.
So I gave it up.
.+@.+\..{2,}
That is, at least one character for the inbox, at least one character for the domain, at least two for the TLD (we assume that TLD-less domains are undeliverable by us). This ensures we don't allow 'a@a' or 'a@a.a', but do allow 'a@a.io'.
dig MX io.
[EDIT: I see that you consider TLD-free e-mail addresses undeliverable; still!]
.+@.+
[*\"32f2@13.31.43.11
they are up to no good and I don't want them as my customer.
Also, according to the standard email addresses supposed to be case sensitive, since the username part refers to a unix user and unix is case sensitive. I work with a lot of email address lists originally collected on paper and of course noone knows that. So as bad as it sounds, part of my sanitation process is to lowercase everything. Noone ever complained. What the standard says and what people actually do is very different.
If it's only people hitting multiple of those conditions, then maybe you could use a scoring function that only rejects users that meet multiple conditions? If the reason for the validation is a character judgment instead of a technical limit, then you could add more conditions too. People with email addresses that contain accented characters probably don't even speak the same language as you, so you could save yourself some headaches by marking those invalid. If you don't like leetspeak, then I think banning the letter "x" could be a good move too on average. "+" is used by a lot of people who want to trace whoever you might be selling your email lists to, so you could ban them before they blow the whistle on anything. Email addresses containing references to political candidates probably aren't owned by savory characters, so you could mark those as invalid too... Now that I think about it, since it's not for technical reasons, then none of this is really specific to emails. You could run passwords and other user content through this validation too.
If the restrictions aren't for technical reasons, then they're just arbitrary tools for blocking users.
Sorry for ranting a bit. I'm just tired of how often I get mystery validation errors from services because something stupid like that it turned out they thought that english works that happen to be sql keywords like "insert" shouldn't be allowed in the subject field, or my friend with an apostrophe in his name repeatedly being turned away from software telling him his name is invalid, and the idea that these rules could be in place just to cause people like us this annoyance ticked me off.
> [*\"32f2@13.31.43.11
> they are up to no good
Well, that one in particular, perhaps — but what about "sam & jill"@ourfamily.invalid? What about john(for sly010)@nowhere.invalid?
> Also, according to the standard email addresses supposed to be case sensitive, since the username part refers to a unix user and unix is case sensitive.
The local-part doesn't refer to a 'Unix user'; it is what it says on the tin: the local part. It's up to the receiving server to determine what it wants to do with case. It's the duty of other servers to preserve the email address as transmitted.
> I work with a lot of email address lists originally collected on paper and of course noone knows that. So as bad as it sounds, part of my sanitation process is to lowercase everything.
I might use the email address as given, and if that bounces try lowercasing. I don't know of anywhere that has case-sensitive names, but it's conceivable.
If I had to guess the main reason this apocalypse is not happening already is because by default mysql indexes are case insensitive, so dude@dude.com is the same as dude@DUDE.com, so people who use mysql never realize this being an issue.
Those of us with firstnamelastname@commonhost will appreciate not getting bills and job offers and such.
Send an email that they need to click on (or an email with a code they need to enter), OR ask the OAuth provider with authority for it to validate it (i.e. Google Oauth for Google addresses, Windows Live for Microsoft Accounts, etc).
The best way to identify someone with an email address:
Store a canonical version of their email address alongside the users email. Use the canonical version when signing-in/identifying and the raw version originally supplied to send email.
This is the only way to not have duplicate accounts for firstlast@googlemail.com vs first.last@googlemail.com vs first.last@gmail.com .
The canonical email is always lowercase, no dots, no + part, no prefix or suffix columns, known domain aliases are normalised to the most common alias (googlemail.com > gmail.com).
I wrote a SQL canonical email func recently (in preparation for Persona shut-down) if anyone is interested: https://github.com/microcosm-cc/microcosm/blob/master/db/mig...
Periods being optional in usernames is something that Gmail started, not a universal rule or part of the specification. Are you going to build into your system all the possible variations for every email provider?
I've analysed my user database and am only doing the large providers for which I have high confidence.
E.g. if I type foo@googl.com, it should be pretty likely that I meant google.com.
This check is done server side, while on the client I just use an html5 input[type='email'] with a required attribute.
For example in PHP I use filter_var function with the FILTER_VALIDATE_EMAIL [1] - while it's great to know why and how to do a particular thing programming, it's better to use a time tested library that is maintained by multiple eyes versus just your own.
The conclusion is sound (although leaves out a discussion of the whether an email confirmation field is at least better than nothing).
[1]: (As a side note, I think the most common explanations for grammar validation are programmer perfectionism and proactively stopping user garbage, such as copy-paste errors or intentionally fluffed fields that will result in a bounced email anyway.)
1. Email validation regexp should be: /.+@.+/ => Tell the user to enter a valid email if that doesn't match.
2. Send a validation email to actually exercise the system
Secondly, this guy is so hilarious.
I clicked to his podcast (where he reads Wikipedia pages) at the bottom and listened to this hilarious episode https://itunes.apple.com/us/podcast/david-reads-wikipedia/id.... His specific style of sarcastic humor (just like the article - is there a name for this style of humor btw?) is rare and hilarious.
2. For immediate feedback, you can reasonably check it's not at at a non-routable IP range, ie. address@127.x.x.x or address@192.168.x.x or similar. However, this check is best done on your email server (MTA policy). It's unlikely anyone would enter this without malicious intent so there's no need to optimize for their use case.
Email validation is simple. You take the full RFC regex, and remove the notations permitting square brackets and comments. You wind up with a regex that accepts exactly 100% of real-world email addresses. If your regex contains "\.(com|org|...)", you're doing it wrong.
If you can't figure out how to do the full regex properly, then match against /^[^@]+@[^.]+\./ - or hell, just check for an '@' symbol - as a basic "hrm that kinda looks like an email address" and send the email. It's really not difficult. The only problem is naive English-speaking people who only deal with latin1 thinking that [a-z0-9]+ is somehow the only valid criteria for a username or hostname.
That would invalidate ceo@cocacola, which is a legitimate address.
Sanity checking them however is often useful for checking form input or data cleansing.
It does seem effective to have someone type the address twice as that can catch a typo fairly easily.
Send your users an activation email.
https://github.com/mailcheck/mailcheck
If you really need to validate, the only way I know how is to send them an email and click a link to confirm.
There is absolutely is a correct way to lexically validate an e-mail address: namely, implement a parser for the syntax specified in whatever RFC is the up-to-date successor of RFC 822.
There is such a thing as incorrect e-mail address syntax: namely, non-RFC-conforming syntax, whatever that is.
You may reject that, and that's about it.
Please don't reject RFC-conforming e-mail addresses.
Anyone trying to provide an email address with a square bracketed IP or a comment is specifically trying to find an excuse to cause drama when their email is rejected. Those exceptions aside, fuck anyone who validates email addresses that don't permit perfectly valid real-world characters like '+', or who whitelist specific TLDs (.com, .org, .net et al).
Beyond the issue of comments, I'd advocate rejecting the use of IP address literals and quoted localparts. Additionally, despite the injunctions on interpreting local-parts, in practice, it is best to treat email addresses as case-preserving: you'd consider a@example.com and A@example.com to be the same email address, but if the user typed in the latter, don't normalize it to the former.
Comments are and should be a real thing. They are, for example, how to use service-specific email addresses when using a mail provider who does not treat + as a folder extension (jack+suzy@provider.invalid is a perfectly valid email address for Jack & Suzy).
FWIW, I tested recently, and Google do exactly the Right Thing™ with comments in email addresses: they deliver the mail.
1: Does it have an @ 2: Is it at least 5 chars long (E.g. a@b.c)
There are also common pitfalls people are doing like empty spaces at the beginning and at the end which should be cleaned up before trying to send the email to " whoever@gmail.com".
There is also an Email standard and when a normal library is able to validate it (and there are free good libs out there) than the effort to do so is similiar minimal but provides an additional support.
What you should do is also to make sure that you are not sending unlimited emails out there. Otherwise you might be missused as a mail relay / spammer.
it's 20 years that people suggest a cool new way to deal with email addresses, I don't even mind listening anymore! :)
— Jamie Zawinski
https://github.com/kdisneur/email_checker
It breaks down into 3 parts that can be used either independently or as a whole: format, MX and SMTP.
Also, since it relies on looking up MX records and then making an SMTP connection to check if the user exists, why not just send the confirmation email? You've already done all of the expensive stuff at that point.
The practical problem is essentially the same as with all the syntax validation out there: It's extremely likely that you are getting it wrong, thus, once again, rejecting valid addresses because your implementation is buggy and behaves differently than an actual MTA.
As for the how so: Because somebody obviously, as is so often the case, just made shit up and implemented that instead of reading the relevant standards and implementing what they say. That's how most of the interoperability problems out there start: People have some passing familiarity with some protocol or data format, and then they implement code handling that protocol or format based on their idea of how things work, instead of what the standards actually say. That's just how we end up with people rejecting email addresses that contain "+" characters in the localpart. I mean, RFCs are public, what more do you need? If you want to implement an actual email address validator, you don't have to pay anything, you don't need to sign up anywhere, you just have to read it, and you can actually build a reliable email address validator, and there is nothing wrong with doing so. Yes, that is more work than just implementing your fantasy protocol. But if you can't be bothered with that, then just forget it and use an existing implementation by people who did care--namely, use an MTA to send a verification email.
register a Gmail address and type in the wrong alternative email.
done. no confirmation required. someone with the mistyped email now have how to reset your password and take your account.
Hardcore.
I worry you're more likely to permanently block a set of users with valid email addresses than prevent a user from making a typo.
The better question: In it's current form, is an email addresses really the best way to do what it is that's trying to be accomplished? (Hint: It's a fax machine.)
I mean, if I have have a phone number, why can't I have an email number? Okay, perhaps not the greatest example. But then again, if a phone number can be switched from one carrier to another, in the second decade of the 21st century shouldn't "email" get the same consideration?
Instead we're talking about regex or some other wonky validation? In 2016? That's just silly.
You can if you want (it is a valid email). But, if everyone had a numeric email it would be a lot harder to catch spam or know if you are sending to a bot. Sure, spammers can masquerade as a friendly name like yourmom313@domain.com now but it's easier to spot names you know than 471871731@domain.com.
> Why can't emails be carried over to other domains?
I like the idea in abstract. Practically speaking, though, this would not only require millions of emails to be purged from every ISP in the world it would also require some kind of creepy global email registry. That defies the decentralized spirit of the Web and would instantly become the Holy Grail of hacker targets.
This means, no email validation, no address verification emails, heck, I don't even have a password confirmation field. One field for name, one for email and one for password and you are in.
If 10% of the users don't trust me with their real, or even throwaway email address, so be it. My goal is not to collect email addresses, but to have users use my application and add some value to their life.
If another 10% of them enter a mistyped email, they will figure it out eventually and change it if they care.