Facebook Stored Hundreds of Millions of User Passwords in Plain Text for Years

It's probably a bit more complicated than that. Usually the things that I encounter have to do with how HTTP requests are logged.

For example, putting sensitive information in a URL that's loaded over HTTPS is considered insecure because many companies have policies where they log every URL that their employees visit. (Think of a password reset link.)

A lot of inexperienced programmers don't realize this, because they don't realize that you can man-in-the-middle yourself, and that most corporate computers come preconfigured to allow the employer to man-in-the-middle everyone.

So, if a password reset link never expires, it means that some guy in IT can own an account that was reset on a corporate computer.

(This, basically, is how they catch people viewing porn on their work computers.)

Anyway, my point is that the problem is probably something where a junior programmer transmitted a password in a way that they didn't realize was being logged.

GDPR may levy fines but ultimately it hasn't done anything to stop managers focusing on new features and sales income.

Check the network tab in dev-tools when logging in to hn for example and you'll see your password in the request body :)

I've never dealt with medical data, but do deal with credit card data so have to deal with PCI. With credit card data we run into customers who send emails, or use online chat, or leave voice mails telling us they have a new card and giving the number. We never ask for them to do this, and indeed everything we tell them says never to send us a credit card by such means, but they do it anyway.

That brings all those systems into scope for PCI, which is a pain in the ass.

We use helpdesk and support chat software licensed from a third party. We had to write scripts that understand its DB schema and data formats and can find and remove credit card information, and keep them up to date as the helpdesk and chat software is updated.

A few years ago, the vendor tried to discontinue the stand-alone version and move its users to their cloud service version, but had to drop that plan when they found out that a lot of their customers were doing the same thing we were, and absolutely could not move to a cloud service unless that cloud service handled PCI issues. Apparently they didn't want to deal with making their cloud service handle that, and there had been no talk since then of dropping the stand alone version.

Do people dealing with HIPPA run into similar issues?

I wonder if it's time to up our game again and go back to that model.

Both are security fails, regardless of the cause of the fail.

And a company the size of, and with the resources of, facebook, most certainly should not get a pass for the "oops, we logged more than we should have" cause. A corp. of their size, and with their resources, should be doing password handling correctly, every single time, no exceptions, no excuses.

This is not an excuse. If you're logging all request data, you need to strip or encrypt sensitive information in that request data. Handling Persistence of sensitive data is web development 101. Just because it's not in a database doesn't give you a pass to leave it unprotected.

This level of incompetency is unacceptable.

All the transport encryption and DB encryption/hashing/salting won't protect you from this kind of logging mistake, but the above would.

P.S. There are ways to make the above even better by adding a nonce that has to be requested from the server before POST etc.

It was easy to overlook, since as you said, it was intended to log something else (failed API requests), it only happened in the case of an error, and it was hex encoded. I only happened to stumble across it out of curiosity.

I don't know anything about FB's infrastructure, but when I was lead I would have viewed leaks in logs as far worse than something in the DB because our DB's were harder to gain access to.

I get what you're saying, but it's irrelevant. Easy to screw up, hard to screw up, doesn't matter; just don't screw up because the result is the same. This stuff is security 101. If you're logging requests then you need to ensure they don't contain sensitive info.

So I agree that these actions are indeed same but different.

They ultimately speak to a failure at Facebook, but it doesn't speak to utter incompetence at Facebook.

Sure, then the question for anyone at facebook would be from the Bobs: "What exactly is it, you'd say, you do around here??

FB claims to have "top minds" in essentially every discipline...

Heck, they are IMO a revolving door to the .gov/nsa/infosec community...

So... I call BS on your statement as "whoops! Easy mistake!"

WTF is it that you'd say you do around here, Mr. FB-Security-Guy??

nope.

there are n types of storing passwords in plain text (including storing them in memory).

you just named 2.

all n types fall under infosec responsibility to prevent, and none are acceptable. there is no pass because logs were less intentional than storing in a table

They're both issues with which any moderately competent engineer would be very familiar. Heck I've seen commercial contracts that specified that audits be done to ensure no password and PII leaks into log files. Everywhere I've worked in the past 20 years that logged stuff conducted audits on a regular basis to check that sensitive data wasn't being logged or in inadvertently stored in a database.

The grey area might be for example when some data gets logged as a blob of HEX that it turns out has a password in it if you run the right decoder function on it. I've seen cases like that though show up in audits -- you see a blob of HEX or MIME and go find out what might be in it.

Anyway, it really isn't accurate to say that this is a kind of <slaps forehead> <doh, why didn't we think of that> thing. It gets thought about all the time.

Its only an easy mistake when you're vacuuming up everything people do.

Edit: You've made this comment twice in these threads. It's wrong.

You have something like four options for fixing it:

1. Every request is responsible for its own logging. This is actually not a bad approach because state-altering requests really need to be logged whereas state-viewing requests are much more optional, they help to try and guess “what were they doing when they ran into this bug they’ve reported?” but mostly they just occupy database rows. The risk is that someone is in a rush and commits something which does not discharge its logging obligation. You can build a system which forces this if you want, “the router will dispatch to your function and one of your function's arguments will be a locally-stateful logger, and once you are finished I will check whether the logger has handled anything, and if not I will log an error. So you should always `$logger->noLoggingNecessary()` somewhere explicitly in the codebase and then if this is wrong it gets caught in code review more consistently.”

2. The sensitive data is used to generate a bearer token and this flow is outsourced to its own un-logged server. You explicitly use the bearer token to construct everything important about the user account in a step before the logging begins, then delete the bearer token from the rest of the request. This flow can actually get really slick: the bearer token can contain the user data, optionally encrypted, with a message authentication code to ensure the user didn't tamper with it: you can then hit a near-empty Redis instance (or a near-empty table) looking for revoked bearer tokens super-fast, since you probably don’t see too much session revocation. So, user data lookup actually becomes unbelievably cheap because it's mostly CPU bound with (check empty key/value store, MAC-or-decrypt, parse body, pass to the handler function).

3. The logging service becomes controller-aware: each controller specifies whether it is supposed to be logged and the logging service just respects that flag and is otherwise global. So it might log that the login controller was accessed, but it doesn't log anything else about the controller.

4. The logging service becomes message-model-aware. This one is actually kind of slick, too, it means that you describe declaratively what sorts of data types are present in the messages that are transmitted to and from the server: and the first thing you do when you get a request is to validate the request against the model you have declared for messages to that request's namespace. So you will have a `validate($model, $value)` function that takes some arbitrary JSON data and a model and returns a normalized version of that data; a natural extension to this traversal that you're already doing (either by returning two normalized results or calling the function with an extra `options={removeSensitiveData: true}` type of argument) will allow you to define in the message-model itself whether the property is sensitive and should never be logged.

This isn't difficult or non-standard. If you are logging all client request/responses full take including auth creds, credit cards, SSN, etc, you are likely doing it wrong, and possibly violating some industry regulations.

There's many layers of wrong if what FB did: carelessly logging production data, letting thousands of employee accessing these logs, and of all these people apparently none of them cared to mention there was a problem here, or if they did it was ignored by management. They don't have any excuse here.

Have all strings printed to logs go through a common checking routine. That checking routine simply checks for the presence of certain hard coded sequences, and raises an alarm if they are found.

Whenever a production system is updated, run a test suite. The tests includes logging in to a test account whose password is one of the aforementioned hard coded sequences. If your system accepts payments, the tests can include a test purchase using a test credit card number that is one of the aforementioned hard coded sequences. In general, for each type of sensitive information, have a test that supplies sensitive information of that type, with that information being one of the hard coded sequences the log checkers checks for.

This won't stop you from accidentally logging sensitive data in production, but it should catch it during the post-deployment tests so you can fix it quickly.

Otherwise, when you use standard C# strings, you don't know when the garbage collector will collect them.

(Granted, you could do similar things with pinning a char[] to a char* and zeroing it before you end the pin, but who wants to jump through those hoops?)

Sure... I bet you eat right and get plenty of exercise, right?

Anyway, that link appears to be best practices when writing cross-platform code.

While you’re at it you can also add some hash puzzle to be solved by the client increasing difficulty with failed logins.

Though if you guys primarily use Git, this does wonders: https://github.com/zricethezav/gitleaks

The only thing I can think of that would help prevent credential compromises is to either implement a company password manager (akin to your butt-in-seat solution) with an ACL, and only accessible on the local network. That shouldn't be too much friction for employees to actually utilize it.

Next is having a secure channel to transmit secrets. That + Password manager has personally helped stop my coworkers in the past from sending passwords in emails, slack messages, post-it notes, text files on their computer, a committed file in a repo, etc.

> crawling randomly through logs for no good reason is a quick way to get in trouble.

So do you get to or do you not?