Namecheap live chat social engineering leads to loss of 2 VPS
postphp.com
postphp.com
I wish there was a way to disable the "customer support backdoor" in all these kinds of services. I've started to deploy full disk encryption to all my servers now so if the attacker does manage to get into the management account the server itself is still protected from single user / rescue mode / etc.
Getting people to do stuff on the phone is easy a huge amount of the time.
Edit. Had some details wrong. It was a reporter at Gizmodo, and basically a collection of data was gathered from various online services (mailing address, last 4 digits of CC) to ultimately social engineer his Apple account and remote wipe his computer. A number of factors were involved, but ultimately he was done in by not having 2FA on google, and not backing up his machine.
Disclaimer: I use AWS extensively. Please do this.
Call 2: "I just locked myself out of the account, can you reset it for me."
Fun story time. I use to play MTGO, the online Magic the Gathering game. Played it from beta for a few years say 2002-2004. Wanted to check it out in 2014 to see how it changed. Failed password reset online, had to call in to support.
The support guy was like chortle what was your security passcode? I had no idea. He tried giving some hints. I said it has literally been 10 years, I am never going to remember. So he went ahead and rest my password. He told me for the record my security code was "I am the nacho king", but I would be prompted to change it at next login.
So could I have social engineered a 10 year old MTGO account? Yes. But without it, I would have been locked out. I was NOT going to remember some stupid passcode kid me set on a game account.
And apparently I am the nacho King.
My wife and I have made, I don't know, 10 guesses over the years and have never been able to figure out what our response was back then.
Questions with fact-based answers are much better. But...I once had a site ask me for my best man's first name (Good! This probably won't change over time!). I filled in "Dave" ... and the site gave me an error "Your answer must be at least 6 characters."
Doh!
Q: what was your childhood best friend's last name? A: pathway-titian-slowly-quiver-kodiak-hue
etc., even for fact-based things like "what city were you born in?" or "what street did you live on in 1995?".
Drop-downs for answers. Just got this on United.com:
The catch-22 of these systems is that recovery questions need to be obvious, memorable and unchanging enough to the user that they are useful for recovery, while also being hard for a third party to guess/research. I feel like for the most part those are more often than not mutually exclusive.
My wife filled one out a few weeks ago where both the questions and answers were selected from popup menus. One of the questions was "What's your favorite summer activity?" Her answer was, "Swimming." Yeah, that's going to add about one bit of entropy to most people's accounts, you idiots.
Another favorite is "middle name of your youngest child." That answer can change over time!
What city was my dad born in? xGU,wT&Yvcn6vr?]#,mE of course.
Little did I know that it's one of those services you need the password (I had written that down at the time as it was temporary) AND these questions that are usually used for password resets.
I don't think that will ever get fixed until I change jobs again.
I always fill these with awkward or absurd questions/anwers that would be amusing if a human operator ever needs to verify them. E.g.
Would you like to go on a date with me?
What color pants am I wearing?
What is the square root of insanity?
Obviously you need to store these in a password database in order to remember them, which kind of defeats the purpose. If I have to choose from predefined questions, it goes along these lines.
Q: What was your mothers maiden name?
A: Why, are you stalking her?
Q: Where were you born?
A: Oh I can't remember, it's been so long!
#$%&+@ is going to be hard to type or speak, just say their nickname is "the frozen one", same entropy, easier to handle
> Use a random but user friendly value
> "the frozen one"
I imagine that may make make your significant other who was previously friendly, markedly less so, if they see your "friendly value". But that may have been your point, as it may be quite a bit easier to remember. :)
But yeah, what you said might happen.
Error: Security question answer cannot be more than 10 characters long.
Error: Security question answer must be one word.
Error: Security question answer must be unique.
Error: Security question error. Please try again.
United Airlines does this.
Their reason is incredibly depressing.
https://mobile.twitter.com/evacide/status/711853927134842880
Infosec is an unsolved problem in so many ways, especially for the majority of people that are nontechnical.
I gave the name, was asked to repeat it, so I did, they informed me I was wrong.
I still don't know if they had a name with a typo in the records, a maidem name, or maybe they didn't even have her name (don't remember telling the bank about marital status) and it was some sort of trick question where I was supposed to answer I'm single (even though I wasn't).
The whole point of this misery was to recover my Steam account to play a few games. Fortunately, Steam lets you recover your account if you can provide proof of ownership (like CD keys of physical copies).
I don't want Google to be the safekeeper of my digital identity.
It's harder online when you don't have the same ability to interact face to face.
Most tech companies (especially startups) don't.
I haven't used PostID yet but with WebID you basically have a Skype video call where you show them your ID.
[1]: https://www.webid-solutions.de/en/ [2]: https://www.deutschepost.de/de/p/postid.html
So KYC as a service is certainly a thing, and I know of two: Tracesmart & Onfido.
The best was when they asked me which model of car I had owned... and listed two cars that I had owned... I could only select one.
These records can be flat out wrong too. The DMV associates a car with my address that I don't own, for example. I think this happened because the owner never changed their address with the DMV. Or someone could have just fat fingered something which gets populated to other databases with data sharing.
Of course that might not be even necessary as there have been reports of people withdrawing money or wiring it somewhere with not even that (but the bank has legal responsibility)
With servers a similar thing. Say my only copy of a database is on my VPS. May have a business value of $50k. Can't really just say no unconditionally. Need some process to unlock..
Banks have the option of you physically going into the branch and identifying yourself with relevant legally backed forms of ID. That would not really work for most online companies.
Other methods which involve sending in copies of ID and/or letters signed by appropriate notaries would fail due to human engineering too because your average tech company isn't going to have people sat ready who are capable of accurately verifying this information.
The other problem is PR: due to the lack of another option the average person who is locked out of their account will instantly turn to twitter/facebook/any-where-else-they-can-post and scream as loud as they can that they've been mistreated by company X. Many other average persons will take this at face value without checking the fats and start avoiding company X, or worse bombarding them with communication in support of the inconvenienced user.
Also, rechecking the ID of a user can be as simple as asking for a new token payment by the same credit card as used by the account. It's not infailable, the CC can be compromised as well, but it should be way better than what we have now.
This is excellent security, as long as they're not sending it to an address you provided over the phone when you requested a reset.
Not always true! Less than a month ago I needed to login to my Wells Fargo account. Unbeknownst to me, they had been doing some 'upgrades' and there were some glitches. After a frustrating period, I decide I'd just go to the physical branch 1/4 mile from my house and get this fixed!
On site, the bank personnel have access to exactly the same system that I did. (At least they knew there were glitches and sorta how to work around them.) I had two accounts, one for a credit card that I rarely used and my mortgage. Turns out, if you have a credit card then the new system requires a piece of information only found on the physical card - the onsite employees couldn't get around and neither could their call-in tech support!!!!
Point is - for log-in purposes - don't assume going to a physical branch will be any more helpful!
Since I didn't have the credit card with me ('cause rarely used) I canceled the rarely used credit card and was able to login shortly thereafter.
E.g. namecheap.com generates verification ticket item requiring valid identification and SSN that Bank of America then uses to verify your identity for $30, and vouched for the identity. Meanwhile Goldman Sachs generates a verification ticket requiring much more strenuous authentication, and the bank charges $200 for (with increased insurance, etc), which satisfies the much higher validation standard the Goldman Sachs requires to authenticate you for your ritrement account with over $X in it, etc.
> "You forgot the password that you've logged in with multiple times... including 20 minutes ago."
That should raise a flag.
People using a password manager might not ever know their password. Funny things happen with password managers where history is missing, changes don't save, keystrokes break things. We can't penalize users who use them.
It's unfortunately a really messy area.
Source: was a password manager in a past life
It's a similar situation to someone who only ever uses their credit card to buy small amounts from their local supermarket. Then suddenly they use it to buy a flight in another country. It might be legit, but it's often not, and should suggest that customer service need to do more investigation before approving.
Depends on what that account controlled.
You don't need to log into your VPS provider's account or domain name provider's account very often, compared to how often you use the machine or domain. But you don't want those getting reset more easily just because you haven't logged into them in a while.
Some people would be happy to pay for a leaked copy of those photos, for use with any other company that would accept only the "face/license" photo as sufficient proof.
I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
There's services on the darknet where you can buy these fake Id picture/scans as well.
I think you are a little confused here. There's no way for Blizzard to authenticate those pictures, you can take literally anyones passport and just swap the name on it.
> Took a heck of a lot more work than asking someone for username/pass in a live chat
This might be true in a world without photoshop, but that's not the world we live in.
>I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
In my personal experience, they'll very rarely insist upon receiving those photos. My battle.net account isn't even under a real name and despite that I've had the authenticator added and removed several times.
As someone who forgets his passwords on a regular basis, I'm kind of glad that there's no such thing as perfect security...
I backup my 2FA tokens using TiBu (encrypted locally and uploaded straight to cloud services). I'm 99.9% sure I'm never going to lose them.
I want an option in my panel that says "do not let me use support without providing a 2fa token".
All of my security questions are passwords. I once had someone at a bank ask "Wait, your mother's maiden's name has a number in it?"
"Wait, you actually answer security questions that any of your friends can guess honestly?"
You want to know how bad it can break down? I imagine the worst case scenario, for so many reasons, actually happened and was mtgox.com. It started out as a Magic: The Gathering Online Exchange (from what I understand) before it became the now infamous Bitcoin exchange that was hacked[1] and massive amounts of money was stolen. I don't know for a fact that old accounts before the pivot to Bitcoin still existed and worked, but it's not inconceivable that they would. One hopes they adopted much better security compared to when they were a trading card exchange (if it wasn't already exceptional at that time), but there could very well have been a time when it was gaining traction for financial type services but didn't have good account safeguards.
1: Or whatever. From what I remember that's a story convoluted and with enough conspiracy theories it's worth a movie.
I guess that's a long-winded way of saying you are probably right.
Then again, bitcoins were worth so little in 2010 that I could imagine the account security on exchanges back then being comparable to hobbiest exchange sites, so the spirit of the comment may be valid even if the specific example falls down. :/
Magic Online accounts can be worth tens of thousands of dollars.
And at least back when I played, if your account got compromised and all your cards were liquidated, the response from customer support was pretty much "that's too bad, shouldn't have let your account get compromised". I seriously doubt they ever added 2FA or anything either.
It might cost 10k+ dollars to make a behemoth account, but I don't think they are worth that much. It's basically fake internet points, I can gain these for free in this very comment.
And apparently I am the nacho King.
What are the chances that you'll remember it in the future? You're already two years into remembering it for the next decade after you found it out again.
Following up I had to reset my passwords a few more times to ensure the attacker's session was properly terminated. They identified the IPs accessing my account but they were all proxy IPs from Europe and Israel so not much could be done.
I still have the support tickets the attacker made, and I feel their support staff should have recognized it as a compromised account (they changed the name to "John Smith", bad spelling / grammar, asking to reset the root password, accidentally "lost" their private key and need password logins and root SSH enabling, etc).
Aside from this incident though I've been pretty happy there, the hardware is competitive, uptime is good and the network is solid which is all I really ask for with a server provider.
I realize there has to be a way to work around 2FA, for situations like loss of device, terminations, etc, but that should have some known policy way of verifying (say, you must send a notarized or local equivalent letter, or appear in person somehow) identity, especially for a cloud service.
(Totally incidental and you perhaps won't see this, but, r1ch, Conflict Crusher was instrumental in 15 year old me learning how to mod TA, which led me to learn how write BOS scripts, which led me to realize that I liked this programming thing, which led me to my current career/way to support my family. Thanks! :) )
Godaddy has gotten really good at preventing social engineering stacks like this. I use 2 factor authentication for my account and customer service can't event talk to me till I give them the code. Don't have that? Need to send them my drivers license and other proof to get the account reset.
Security is almost bomb proof with IP restrictions, GPG keys, 2FA and one little checkbox in their settings I like a lot: "This setting allows you to authorize or disable password resets from the login screen."
I think a good practice is to try to social engineer your own account and see if you end up getting it.
1. The credentials were resent to an already compromised email account
2. This is an isolated case
3. Established procedure was not followed
4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved
5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups
Why have a procedure if your support doesn't follow it? Even if you have a procedure, everything falls apart when it isn't followed. This is the same as having no procedure at all.
Note: I have no direct or indirect relationship with Namecheap at all.
Wouldn't it make sense that support staff can only generate and send out password reset mails if the PIN/password has been entered into a form? I don't know the term for this - like "coded procedure".
In this case, the support staff wouldn't even needed to be trusted in the first case.
EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently.
EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology.
EDIT3: congrats to Tamar for being promoted to a Namecheap executive!
The opposite of what I'm suggesting is that people - individuals/companies - do not look after their own backups. That's a dangerous precedent.
This is what they read as the company's response to this loss:
"Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass."
Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand?
Now consider this reinterpretation of the statement:
"With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."
Although backups are #1 item on any list of best practices, making an easy, and tested, implementation method would be a good practice on your part.
Unless you sign up for a managed service that claims to include backups or whatever, you are responsible for your own backups. What's controversial about that?
I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid? Saying so doesn't mean the the mugger isn't guilty or that what happened is right in any sense. It's just acknowledging reality.
Yes, this is the textbook example of victim blaming. Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged. I am admittedly not the best at describing this because up until recently I had the same thought process as you. I would encourage you to find better explanations than what I can offer and be willing to have your beliefs challenged.
Then "victim blaming" is a meaningless concept and we should quit using it. Because if I choose to do something stupid, I do bear some responsibility for the outcome, even if somebody else violates my rights. That doesn't absolve the other party of course, which is my point. That is, you can blame the perpetrator of a crime while also pointing out that the victim could (possibly should) have done things differently.
Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged.
It isn't "blame" for the actions of the other person. Why wouldn't you point out the stupidity of knowingly putting yourself in a dangerous situation?
You can say a victim is stupid without giving any vindication to the person in the wrong.
Namecheap are saying "be responsible for your backups, but yeah we screwed up on our security policy" - They are seperate things, that the victim here has conflated, but are seperate problems (in regards to namecheaps offering).
In other worlds however, the problem is usually too widespread. You might get a lot of attention, comiseration, etc. but in the end, being reckless goes against survival. People who point this out should not be shushed for pointing out what you need to do to survive.
Its amazing to see that this "victim blaming" mentality is growing in Brazil. Violence here is out of control. You might get mugged/shot/kidnapped for no reason, or not displaying any wealth. Having been kidnapped myself, and chatted with the kidnappers, they do look for signs of wealth before pouncing. Therefore, yes, the victim does has an ounce of control over their risk and it's not wrong to point that out.
It does not solve violence, and attackers will just look for other victims regardless of their reward estimate. However, would you tell your children not to not show affluence/vulnerability in shady places just because you don't want to "victim blame"?
Also, it's stated in the knowledgebase that it is advisable to set up server backups of your own if you do not have a managed server: https://www.namecheap.com/support/knowledgebase/article.aspx...
Even my managed services have offsite backups. Better be safe than sorry, I always say.
I don't think the best way to respond to a public vent is "Here's what you should have done instead". Responses might be technically correct but they lack empathy for the customer.
For example, let me give you a look at what my Windows hard drive looks like.
My important files are stored locally, on Dropbox, and on CrashPlan. Some is also on Google Drive. I also run an offsite backup of my own to another local Linux box.
Don't make this specific to Namecheap, @kelukelugames. It's always smart to have good recovery systems in place. If you care that much about your data, you will protect it at whatever cost.
So yeah, I repeat, better to be safe than sorry. Your mileage may vary.
Usually there are backup options included in the plan for upsell possibilities with these kinds of providers. Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.
An example would be if some service stored credit card information temporarily while waiting for transactions etc. to process but then purged each record after two weeks later. A compromise of the backups containing, say, weekly snapshots could then contain 90% of a client's ever-stored financial information whereas a compromise of the main site might only reveal a couple percent of them.
In reality though, more companies do this than should be allowed. I worked for an ISP in a previous life and even on the super cheap shared hosting there were companies that were making a decent turnover and then using the cheapest possible hosting for their email/site. The quantity of these companies was a significant number too.
Especially when they were kicking off on the phone due to inevitable maintenance/downtime. Trying to appease customers that turn over 10 million a year and pay £5 a month for hosting is a bit wtf. You pay for what you get... that's no different in hosting.
They never spell this out for you though, usually they imply that their service is just as good as their pricier rivals. As a result, many people get burnt before they get savvy. Some never get savvy, they just get turned off to the industry.
Not sure what the alternative is. I suspect a company that did clearly spell out their pros and cons would risk having stunted growth or go out of business entirely.
"If it's too good to be true, it probably is"
and
"Cheap, Good, Fast - pick two"
To be fair, your third edit was only for me. And that was the only comment I was replying to.
As I said, we are working to respond to hundreds of comments across dozens of platforms. I certainly respect your distaste in the more rushed responses in order to address all of the deluge, and that is why we were also simultaneously working on a longer and more thoughtful response that speaks for all of us at the company via that blog post (in a far more emotional tone).
It certainly is difficult to envision the challenges of responding to dozens of responses if you're not in our shoes. But I genuinely thank you for the feedback - and we're noting this (as well as the feedback all have been sent to date; a lot of that was factored into policy adjustment and our blog response) for handling it differently next time.
This is not the correct solution. What's to prevent the next new person from making the same mistake?
If it shouldn't happen, don't make it possible to happen. Put in place a technical solution that doesn't allow it happen. And if there is some special case where it still needs to be possible, make it that it needs a secondary signoff from a senior team member.
People will always be fallible.
And how many people at Namecheap do you think aren't aware of this by now?
But yes, technical solutions should go in but those take longer to implement. Among other things, it seems to me that re-prompting for the account password might be a good idea before any VPS reinstall/reinitialization that's going to wipe an existing VPS (not that it would've helped much here).
---
Disclaimer: I'm [not] CIO @ Namecheap
We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers.
In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing.
We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here: <link>
I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.
* Actually apologize in a human way
* Show empathy by identifying the impact of what happened to customers (not your impact internally)
* State action items that you've created, even if they are just in 'evaluation' state
* Indicate that the specific incident in question is being handled outside of this forum
* Take responsibility for things even if you shouldn't "have to"
http://blog.statuspage.io/why-public-apologies-suck
Some important bits:
>4 PARTS OF A BAD APOLOGY
- Justifying the offending actions or words.
- Blaming the victim.
- Making excuses.
- Minimizing the consequences.
>8 PARTS OF AN EFFECTIVE APOLOGY
- You actually have to use the words I’m sorry.
- Acknowledge that you messed up. (As in, “I take full responsibility for my words.”)
- Tell the person how you’ll fix the situation.
- Describe what happened, but without foisting the blame off on someone else.
- Promise to behave better next time.
- Make sure the person knows you know exactly how you hurt or inconvenienced them.
- Much like the first rule, it’s important to use some version of the phrase “I was wrong.”
- Ask for forgiveness.
Overblown/fake apology is not very informative - it's hard to say what exactly can you trust in it.
It sounds like you are confirming that this incident did happen and it was your fault for not following your procedures.
I am not a lawyer, but since there was signification loss, it would probably be in your best interest to offer better reparations.
OpenDomain has several domains that are on NameCheap - I will transfer them immediately since it appears you do not care about customers.
We have apologized, admitted mistakes, and made tremendous internal change to move on for the better. We would not do that or even post here if we didn't care.
You also didn't mention all the terrible things the OP pointed out that someone can do with just your password even when 2fa is enabled.
However, it's relevant to the story because there's a huge difference between sending a password reset to the email already listed on an account vs. resetting it for any random person who starts a chat.
This doesn't excuse their other issues, but it makes the customer support rep's behavior a bit less awful, even if they still violated protocol.
And although it's not you area, can I just say that Namecheap's website is just way too slow since the redesign? I appreciate that you even did a redesign, but for some reason it's one of the slowest websites around. I don't know if it's because of the large images you use on your pages or what's the problem, but I suggest you fix it. It may be losing you customers. A web services company's site should be snappy.
You could have just as easily lost all the data in an accidental way, with no malice or 3rd party involved.
That said, I do empathize, and it's disappointing that a major player like namecheap would be so easily socially engineered.
Self-managed a customer is responsible for their own backups. Just like with DO and that full server loss a couple of months back.
Am I missing something, or is Namecheap saying they didn't do anything wrong?
https://news.ycombinator.com/user?id=pg
https://news.ycombinator.com/user?id=dang
https://news.ycombinator.com/user?id=sama
Why is it so hard to put info in your profile or to put a footnote in your comments for the newbies? Would you have your business act this way? Reply to a person's inquiry and not say who you are and what you do? Of course not.
"we offer service X"
could be from anybody trying to say they offer the service, not obvious at all it is a representative of Namecheap
> Disclaimer: I'm CIO @ Namecheap
It's a $30 to $75 a month upcharge on top of the base VPS price. I don't think the OP was paying for a managed VPS.
Edit: Apparently, only the $75/month package includes backups. The $30/month package does not. (http://i.imgur.com/Iy7iacH.png)
Are those backups purgeable from the account control panel?
Honestly I'm not a huge fan of same-provider backup solutions. It seems like asking a fox to guard your sheep.
I think a better suggestion/wording (depending on your intent) re: backups would be "don't operate without independent backups".
If you give a shit about it, back it up.
Yes, you should always have more backups than you need. But wouldn't you be moving to a different provider after something like this anyway?
This means there can't be any "impedance" mismatching that can be used from one service to another. For example, one company gives out the last 4 digits of the credit card, and the other uses the last 4 as security info.
What we need is a uniform security standard and training for ALL customer support personnel so that you can use Apple to break into Amazon, or Digital Ocean or Namecheap. The staff need to be trained to never succumb to social engineering ever, and in fact make it impossible for 1st line support to reset anything. Have any security information get passed up to second tier support who are extremely well-trained. Etc.
And have this standardized so that there is incentive for customers to look for this certification so that we don't have to keep suffering the same mistakes over and over again.
You're joking right? He even fully admits that he did what they accused him of doing.
I'd definitely agree that the solitary confinement was cruel and unusual. I'd also agree that the law wasn't mature enough when he was charged so he was charged with proxy-laws.
But ultimately he did do what they said he did, and some of it was pretty messed up. He would definitely be charged today with computer crimes (or generic crimes) for many of his exploits at the time.
(the book is great, by the way, highly recommended)
That is like showing someone "here's a lock and what's inside of it."
In time, someone will pick that lock.
Uniformity is what you don't need, nor would you want to know the nuances of how security and privacy are handled at a company so that you know exactly what holes need to be exposed.
You can't standardize security. It's way too risky.
A standardized process design could be carefully examined and improved to plug the holes, so all you're left with are implementation bugs. You'll never get there with a thousand disparate processes: they'll have design and implementation bugs, as well situations where system compromises data used to secure another.
Plus, standardized processes would allow implementation of more expensive processes. For instance, you could have a higher-grade fallback "prove who you are" process that involves going to some designated office in-person with all the right documents. Not even Google would pay to setup such offices in every city, but if Google, Amazon, Online Banks, etc. all would use it, it might be possible.
I know, this is an issue some would disagree with. I do not think it's safe to standardize at all.
See encryption. A secure crypto mechanism is not vulnerable to disclosure of its mechanism. The key is the secret. The mechanism is not. Having a secret mechanism that relies on its own secrecy means your main weakness is someone explaining how it works to the outside world.
RSA is perfectly well-known. So is rot13. rot13 is crap, disclosure or not. RSA is not as crap, disclosure or not.
Your method of security through obscurity simply doesn't work because hackers will figure it out and exploit impedance mismatches between vendors.
There is no reason for Namecheap to have a human in the loop for this, at this stage.
> I’m pretty careful to use 2FA for any service that I consider important
I'm surprised no one else has mentioned no 2FA for the email. The email being compromised opened the door to this happening.
https://news.ycombinator.com/item?id=11480221
You should not be able to overcome 2FA with social engineering wtf!
It doesn't help that the front door is securely locked when the back door is not! :-/
> The VPS panel allows full serial console with only a login/password (no 2FA required or possible)
Yea that's because it's a serial console, if you want 2FA or something then that's a matter for your operating serivce. A serial console is literally like you're plugged directly into the machine.
> They send out your VPS panel login/password in plain text emails when you sign up, and when you reset the password. So if you ever failed to delete one of those emails completely and someone gets into your email…your totally screwed…
To be fair this is pretty standard. It's your job to secure your passwords once they've given them to you. If they're storing it in plain text then you can complain but this basically sounds like you're complaining that they're not encrypting emails. Sure they could only show you it once when you boot it up. But since this action was done via customer support they would have to give you the password some how. To your email address the most secure other than the chat which can be by an attacker like it was in this case.
> VPS can be irrevocably wiped within seconds without any prompts or confirmations just by the click of one button; whether the server is turn on/off it doesn’t matter.
This isn't a security issue. A UX issue yea, but it's not even that big of a deal. It's in an area you won't be that often and where you know you're doing admin related thing.
> They keep no backups, even to cover hardware or security failure.
This isn't a security issue. It's your job to back up your stuff not a VPS provider.
> And of course the icing on the cake is that they ignore 2FA and are willing to send out your username/password to anyone that asks.
Yep. Pretty valid.
This practice has always bothered me. An expiring link to reset is much better.
Which, once the machine is up, is basically just exporting getty and exposing the system login prompt. Why couldn't the system login prompt require 2FA. Mostly for historical reasons I'm assuming, but just because it's serial doesn't mean it isn't just some software on the other end.
1) Offer an option to opt-out of all automated account recovery. If set, no more email resets, support PINs, or similar. This would be targeted at people truly care about security and have no issue with "forgetting passwords" (i.e. you use a password manager and you're not an idiot about backups).
2) Offer in-person, manual recovery. To participate in this you'd need to pre-register with full contact details (name/address/etc) of the valid people who could use this feature. The person would have to physically come to the office of the company, present two (or more) forms of identification. To add further security, you could add a mandatory wait period between initiating a reset and it taking effect (ex: min 7 days). That way a combination of fake ids and social engineering could (in theory) be stopped by getting an alert that "You initiated a manual reset of your XYZ account. Did you actually do this??"
EDIT: For #2 you could also add a non-trivial fee (say $500) that would need to be charged and cleared in advance of the person showing up.
There's also a checkbox that says "don't ever recover this account" (i.e. the "I have a password database on Dropbox") checkbox. Checking that box actually disables password resets on the admin interface, so your account is pretty much dead if you lose the password.
If you announce sending the reset letter there is also time for the account owner to prevent the reset unless an attacker manages to isolate them from all notification channels.
Sometimes you get what you pay for.
I just wish that their DNS updates were push through faster.
And presumably they wanted you to send those photos to them as an unencrypted email attachment, right?
Edit: Since I'm getting some downvotes I'd really like to know how one could possibly argue that asking for ID scans is better than email resets. You can't really forge the ability to receive email at an address, but you can very easily replace the name on an ID scan.
An attacker can't just pretend to be able to read your email, such ability is too easy to conclusively prove. To be able to read your email they need to hack you somehow.
But for a fake ID the attacker only needs to throw your name in a PSD and they're good to go.
As for fake IDs, yes, it is certainly possible to create them. But when it is so much easier to socially engineer your way into another service like Namecheap, it creates a disincentive for going after Gandi (and other similar hosts).
No security measures can be foolproof, possibly short of sending someone to your home to take a DNS sample, but at least they're trying for a better solution.
That's why nobody has ever used a fake ID at a bar!
> but in germany the fake itself is punishable by law (up 10 ten years).
https://dejure.org/gesetze/StGB/267.html 5 years.
But producing fake scans isn't covered by this law, scans aren't even an official document. In fact, it is illegal for a german company to ask you to send them scans of official documents.
> You'd also need much more information to create a convincing fake id scan of your intended victim
To fake a good enough passport scan you'd need your victims name. That's all the rep is going to have.
I sent the ids by fax (yeah a few years back, I still had a fax machine).
I thought asking for id's + phoning on the number listed in the whois database was a good cross check, especially back then.
Use 2FA on your Namecheap account
Maintain a sensible backup policy
Store your passwords in something secure like KeePass
So what expensive provider do you recommend instead?
What? Why do they do this?
And no, it wasn't related to registry rules or transfers as other users have suggested.
I personally haven't had this happen to me. I've hosted dozens of domains with gandi, under a variety of different TLDs, and can only recommend them.
Cons: Slow website. Bad UX all over the admin/purchasing interface. Feels like they're not doing anything to improve that.
Sure! Gandi received an abuse report regarding someone using one of my domains to scan for open dns resolvers. I informed Gandi that there was nothing I could do about this and expected that to be the end of it. Instead, they suspended my domain and started demanding that I send them ID proof.
As I needed the domain back I sent them a redacted photo of my id card, after which they demanded to see the full id. I decided to terminate my relationship with them.
I feel that this was absolutely unacceptable and likely unlawful behaviour from them as they had absolutely no need for that information. This wasn't a whois dispute.
I do agree with more login forms needing to support 2FA. At this point I wish almost everything did. It is a bit more hassle but is easy to manage for me at least.
This is the same for DigitalOcean. I'm always amazed that clicking "Rebuild" or "Delete + Scrub Data" doesn't require _any_ confirmation at all.
Is it? Does that mean that my ability to reset your users solusvm passwords with or without 2fa constitutes as a 1337 0day?
Hey BTW, remember that time you got hacked through your support site and didn't tell anyone?
What do you use to tell whether a chat session is a genuine user or someone successfully using a social engineering attack against your chat operatives? If the answer is "nothing" then you can't know if this is an isolated event or how many of your chat sessions go without a glitch.
First of I can social engineer one of your staff. Regardless of how much you train them. I could also bribe your staff or try to get you to hire a plant. Yea that last one is far fetched but just making a point that as long as someone can manually do these things someone will.
No, it is never okay to compromise security just because "it's good when you forget your password". There should be no way around this for any reason; a bypass via SE or any other mechanism is a failure of the company, end of. If you forget your password or you do not have your 2FA/security questions available, tough, you should lose access.
For the sake of "workarounds for legitimate users", that just translates to "a security hole".
Special pleading is a logical fallacy and if it works on a support rep, that rep has failed at their job.
No one is arguing that.
[1]: https://www.namecheap.com/support/knowledgebase/article.aspx...
Its been months since I wanted to write a detailed summary, but the notion that "namecheap is hackers best domain registrar" is not valid anymore!
About year ago I noticed DNS changes on many of my there-parked domains. Upon reaching via Chat (no phone support so that angry customers cannot vent off) I was told that they cannot help me cause Im not the owner of the account! Upon full verification even with CC on file and telling them purchase history going back to 2009, I was still denied the access. As it turned out, all hacker needed to know is my public WHOIS info to take over my account!! That was insane! Only continuance of threats from my side that I will plaster it all over the net made them change their mind, which again is a breach of trust - what if I was actually the hacker??
What really made me start moving domains to NameSilo (Im not affiliated) is that upon doing a thorough research, I found many cases where Namecheap gives up on fighting for peoples domain! I seen names like nanotmz where company was building some sort of magnetic devices and TMZ came in and threat to sue Namecheap if they dont shut the domain down. That's where I found similar cases for NameSilo and learnt that they stand their ground and would not give up on your domains, even if are threatened with legal action.
I'm out of Namecheap completely as of last month with last SSL expiring.
More bluntly, what is it that you think your current registrar is lacking?
I read a few comments on Gandi and support of GPG keys. I'm guessing this is what you're referring to: https://wiki.gandi.net/en/gandi/documents
Customer support should not be able to see anything about these accounts except for a reset button. I do expect to be charge a fee for a reset if I need to use it. This would need to be rate limited to prevent people from dosing an account through it.
If you want them to go thru the trouble of mailing something, at least require it to be a letter. Inside an opaque envelope.
A postcard is the exact opposite of something you want to use to send sensitive information.
Let users chose if their account can be recovered through a password reset form, let users leave 'secret memos' for support which can't be viewed on the site later, let the users decide if their accounts can be altered in any way by support staff, etc, etc.
All I had to do was tell the chat operator I was having some issue with .htaccess and .htpasswd until they offered to delete it temporarily. In the few minutes between them deleting it and reuploading it, you're free to do whatever you want.
I reported this and was told it was resolved (I guess with new policies).
From 2014: http://thenextweb.com/apple/2014/12/08/lost-apple-id-learnt-...
They no longer do this: https://support.apple.com/en-us/HT204921
Fair enough. There's a subset of people who want that and would agree to that beforehand.
But, and here's an important "but", do not just display a checkbox and an Agree button. Instead display a sentence something like this:
I UNDERSTAND THAT I WILL PERMANENTLY LOSE
ALL ACCESS IF I FORGET MY PASSWORD AND
MY RECOVERY KEY
I UNDERSTAND THAT I WILL PERMANENTLY LOSE
ALL ACCESS IF I FORGET MY PASSWORD AND
MY RECOVERY KEY
After you've done that, if a customer is later locked out, that customer will STILL be mad at you. It will be the providers fault that the customer can't get access. No matter how explicitly the customer agreed to the terms.
That's human nature. I don't care how explicit you make it, you simply can't win this one.
There will always be rare occasions such as this, but considering how many customers Namecheap handle, I don't think we should be seriously concerned. I'm pretty confident lessons will be learned.
It's a level of security-through-obscurity, yes, but that doesn't mean it's wrong. It means you can keep that address monitored well. You could make any activity on it send a page, for example.
"This was an isolated case and procedure was not followed. I can assure you we have addressed it so it won't happen again."
Namecheap is obviously to blame for the compromise of the VPS, but failing to secure an email account which can be used for password resets is even a bigger fail, IMO.
I also had one of my VPS attacked recently, and I feel for you.
But the name namecheap says "cheap". Maybe they are indeed cheap? I'm not sure the same would have happened with say HE. You pay, but you know what you pay for and get in return.
Personally, I am thinking about moving from a "manually setup" distribution to a "no ssh but deploy", so as to ease reimaging in the future. This way, if a server is compromised, all I have to do it to start the install of a new one.
Any suggestion for tools to do that with Debian distro? (yeah I could write a shell script, but I think there must be better tools out there)
Well, conceivably the second factor could be used to generate crypto key material which is used to decrypt/unlock one's record, so without the second factor even support couldn't read & edit one's record.
Nothing can stop support from deleting & recreating a record though.
If you write apps, package them as Debs. If you need to configure other Debs, make config packages with config-package-dev [1] from the DebAthena project.
Create a metapackage that depends on your software + config packages, and your setup process just needs to be "add private apt repo, apt update, apt install <metapackage>".
2. not deleting the mail with the login information
3. not having a backup
Yeah. This was just as much your fault.
I had a similar experience. IIRC TransUnion was the problem one. They wanted something like 3 credit card numbers as part of the identification. But I only had two active credit cards. Fortunately I was able to find an old cancelled one in a drawer and they accepted the number!???
The problem is that you're not their customer, you're just an irritant that the federal govt demands that they give "free" information to.
They treat their real customers much better. E.g., go to a used car dealer. Give them your SSN and they'll have your life story in front of them in about 10 seconds. They get good service because they're paying for it.
$CUSTOMER calls in, their nameservers are down and nobody has the account password. Do you think the management at $CUSTOMER is going to accept "hey we need to wait 6 hours to get our site back up because namecheap wont allow us in"?
I don't see any perfect answer here. Ultimately you need a way to recover your account when you've lost all of the "somethings you have" and you've lost your "something you know", but then that allows a social engineer access to do the same. So let the user decide during sign-up.
I don't know if it's changed since this happened, but in the early-ish days my friend's phone broke and he lost his ability to generate 2FA codes. Blizzard was happy to remove his 2FA once he had made a photocopy of at least one (maybe two?) forms of ID and (I think?) some evidence he owned the credit card paying for the account. Once he mailed that in to Blizzard hq, some human confirmed the info and they removed the 2FA. If Blizzard can do that, and they're protecting MMO characters, certainly other providers can do so as well (maybe for an increased fee as I realize it's more expensive than online chat).
IIRC, people hacking into Blizzard games resorted to compromising users' computers and capturing the 2FA codes in flight - then logging in and changing the credentials before the user could react. That's a much higher bar to clear than the one here.
The only solution is to remove these sort of powers from your general tech support guy and let them in the hands of a few highly technical, well trained, well paid staff members (presumably managers?). Of course, I made the blind assumption that your average support staff member is not paranoid enough, but based on my acquaintance with a few guys in the business it seems to me that the salary is not high enough to expect well trained technical stuff doing support.
I might be wrong, and in that case I would gladly know which companies employ such well trained staff, so that I can move my servers there.
"Please note that without your phone, the recovery procedure will take 3 to 5 days". hopefully, this means that there will be many many checks to avoid social engineering
I find it hard to believe Microsoft of all companies has outsourced two factor authentication.
(disclosure: obviously I work for Namecheap.)
Also checked: if you pay for their backup service, nobody can delete the backups, so those backups are even safe if compromised.
Of course this also requires that you should never be able to add a number using the phone though, but this makes sense, since they can just say: "To do that you just have to sign in and click on..."
I read through this looking for the part where two Vice Presidents from Namecheap were fired over the incident.
If it's "Whats your mother's maiden name?" and they let you reset it in the browser, it's a bug.
But if they send you an email (in my case to Gmail, that has 2FA turned on), then it is a feature, because then you'd be required to either 1) intercept the recovery email (and get the password reset URL) or 2) know the format of the password reset URL and just happen to guess mine after brute-forcing every possible link (assuming there is no timeout for the URL or anything else like that).
Have the customer provide an SSH/GPG public key, and store it with the account.
When a password reset is requested, encrypt a random string using said public key, and email it to the email for the account.
An attacker who may have breached your webmail is then reasonably unlikely to also have your private key to decrypt the string.
Follow the link (which didn't necessarily need to be encrypted) and enter the string you decrypted to reset the password.
On a related note: do any/many sites with 2FA, require the 2FA code to do a password reset?
With a 2FA code, you either a) use the same code they use for regular logins, or b) require them to find a way to securely store the (T|H)OTP secret and then add that information to a 2FA app when they want to do a password reset.
I realise the pubkey concept is more than most people would bother with (or even be able to get through on their own), and I think the first 2FA option is definitely better than no extra security at all on password resets, but my thought was about increased security for those who are particularly paranoid/security conscious.
Doesn't this just move the problem from "I forgot my password" to "I lost my private key"?
If you enable 2FA on an account with a service, and subsequently lose access to 2FA otp's (e.g. phone lost/wiped/etc) and lose access to (or never kept) the recovery codes, you generally lose access to the account.
This is similar, but with the benefit that you can keep the key secure by default - i.e. put a passphrase on the key and then store it wherever you like.
In reality, if you fail this "recovery" method, my next suggestion would be a billing based one (i.e. talk to a human, get confirmation of previous invoice details, what is being billed for, how it's paid for, etc)
Yes, I work for Namecheap, but that's probably implied by my comment.
Would anyone here seriously expect that someone in control of their email wouldn't be able to take control of associated accounts?
I'm traveling constantly. I always have a different country's SIM in my phone meaning I can't receive SMSs to a static number.
Is SMS only 2FA acceptable?
* If they happen to be using authy for 2FA and you have the Authy app on your phone, it will use that instead of sending an SMS. You could also just have it send to Authy's Chrome extension.
* Consider setting up a Google Voice number to receive the SMS.
Google voice ok, but given google hasn't updated google voice in like 3 years I expect they'll announce it being discontinued soon.
Any other options?
Which VPS providers do it right?
Which are safest from social engineering attacks?
I understand that there should have been more layers beyond this and all, but really, what is the point if you're vulnerable across several OpSec levels?
The domain registration process isn't automated from my experience but it works well. However, I think the meta is (I think many people will agree) to not mix domain and hosting with the same provider. For example, if you get your domain from namecheap, you should not do hosting at namecheap. Therefore, the argument is that if you want to buy a domain name from Namecheap (as they're pretty decent), you shouldn't do hosting there.
Sorry if I sound like a prick.
Our KB platform is getting some attention as articles are improved and then the UX will be overhauled. We have work to do here and we're doing it.
Just a few weeks ago I was getting a domain set up with Amazon SES, and one Daria P. helped interpret Amazon's docs to get it verified with them, and explained how I was using the dig command incorrectly to inspect the domain's settings. It's rare you see a support person do that at any company.
And, of course, they did ask for a support PIN.
This isn't new behavior for us. I've been at this since 2009.