Facebook engineers: we have no idea where we keep all your personal data
theintercept.com
theintercept.com
Literally everything predates everyone who lives there. Generations and generations of original designers, architects, and laborers have arrived, plied their trade, and moved away. There are people who are experts in certain parts, and who can build a new skyscraper at any given spot, but it is just layering and organic growth.
The emergent complexity of centuries of being lived in and adapted belies easy understanding.
Large tech companies are similar. You just can't understand how "it" all works. If you were to build it from scratch, perhaps you could, because it would be simpler and clearer, but nothing was made with the current state in mind. It evolved and adapted over time.
So reading this, I am not surprised. I think you'd get the same answer about many other aspects of data, code, system history, etc at any other 10+ yr old tech giant.
(There may be other things taken as seriously I'm not thinking of, but compliance and security are the two I've seen drag people out of their beds by their ankles)
- reality
- the socially acceptable fiction in the report
the exercise would be to create an output that allows people to pretend we know where all instances of "log4j" were used, or that sufficient depth and resources have been expended on such.
but in a sufficiently large and complex organisation, we (and by we I mean technically able, knowledgeable or sufficiently aware individuals) know that the content of such is likely to be as factually true as reading a historian's report entitled: "enumeration of all points in past century where cutlery has been used".
It is very important that we are precise here: they are trying to come up with a story to make this about technicalities, and it is actually about their priorities. People who've done large-scale engineering need to be clear that even within a large organization, things do get hunted down if and only if the organization prioritizes them. Measurements and control mechanisms are built. Systems, social and technical, are designed.
And if they're not, Facebook can't honestly say oh but it was so hard it's impossible, we're just too big. We're not talking about detecting I-know-it-when-I-see-it obscenity or whatever social thing people hope AI will fix for them, we're talking about knowing how data is being plumbed from one place to another, and where it lives when it's at rest. It's an embarrassment that this is being presented as a Hard Problem.
I think it's possible to get too abstract in discussion with an idea of "documenting a system" since that can of course shoot for a level of detail that can't be maintained. Here is a practical example.
https://en.wikipedia.org/wiki/Personal_Information_Protectio...
When this law was to come into effect, every company that does business in China had to audit where they were storing all Chinese PII in order to figure out what they might need to change. For large companies, this involved making everyone perform this audit for their own little fiefdoms.
Being as this wasn't the kind of thing one can have lawyers go challenge in court, there were two options:
1. Actually do an audit (and if you're not dealing with exceptionally poor engineering, this should not have been as hard as keeping track of dependency libraries), rebuild everything from scratch (wasteful), or nuke absolutely anything that might have a chance of having Chinese PII (...unlikely from a business perspective).
2. Lie (internally / externally) and thereby break the law.
People in these comments seem to love the phrase "socially acceptable fiction" and I want to be very crisp that we are dealing with a clean dichotomy. You do the audit, or you lie and break the law.
Yes, I understand there are some among us who have so little integrity that they'll shrug at the latter option, that they'd fake the VW emissions test, that they'll cheat and embezzle and tell themselves that it was all inevitable anyway and anyone in their places would do the same. But this is not actually true. We would not all do this. Pretending this is normal variance gives cover to scummy people.
I therefore object to the root comment's characterization of this kind of thing exceeding real understanding as a natural consequence of scale/time. Yes, in terms of how it can come about that no individual knows this stuff, it's accurate – but only in the absence of the institution itself caring to pay attention. I object to succeeding characterizations that it is an inevitable attribute of social systems that meaningful control is impossible, because even educated and carefully hired professionals can apparently only be expected to behave with the ethics one might expect of a sixth grader who would rather let Chegg write his book report.
From the article, this is what FB is saying:
> “We do not have an adequate level of control and explainability over how our systems use data, and thus we can’t confidently make controlled policy changes or external commitments such as ‘we will not use X data for Y purpose,’” the 2021 document read.
This may be accurate about Facebook; I would not know. What I do know is that it is not an inevitable consequence of scale and lifespan.
Disaster recovery/business continuity? Absolutely, you need it documented so that someone 'off the street' could be helpful.
Implementing yet another service gateway? Probably going to take a lot of conversations with people familiar with the streets involved.
Auditors won't care until it's realized. They'll want things like interconnection diagrams -- painting a picture so they can follow your adherence.
We have very high compliance requirements where I work -- about half of my time is spent on it.
They're less worried about how the bread is made, more that you can keep making bread the way you say you do.
Here is one conversation I’ve heard: “We stream our logs [full of PII] to a central repo. Do we need to include that in our docs?”
“No. It’s just logging. It’s a standardized tool we use. It’s probably assumed. Just include the database and known clients.”.
In my experience with the above, people get paged until a "socially plausible fiction or resolution" can be created.
There's several versions of these, one of which is to engage in enough rituals and processes so that we all pretend the task has been completed and the requisite outputs and points of reference exist. An archaeologist knows they don't and they can't and that we lay on bedrocks of speculation, history, ignorance and myth, but we don't have to worry too much about the fiction we create because we also know that disproving such mechanically is also extremely impossible/unlikely.
And we have a host of other social functions to get out of the cases where it could, such as:
- throwing people under the bus/human sacrifice/ finger pointing
- contrition
- plausible deniabilty
- continued non-compliance and non-enforcement
- bribery
- bullshit jobs
And you will trust the reference like if it perfectly describes the reality. Because, really, there is no other way to deal with it. You can test a thing or two and improve it, but you can't test for the absence of necessary information.
The Facebook situation is inexcusable. But no company has perfect knowledge about this either ("perfect" here carrying a lot of weight).
For way too many companies, "IT" is just another cost center that doesn't bring in any profits and so is barely kept at life support levels. And when something inevitably happens, it's rarely the C-level that gets to go to the unemployment office, it's more one poor soul who has complained for years that he needs more staff and budget to at least get the most egregious bullshit fixed.
I mean, we have in all other forms of engineering. Why is software different?
These are all artefacts of software development as we practice it, rather than truisms.
> In practice, software is more like a craft than an engineering discipline.
Maybe we should reserve the term software engineering for the applications of software development where rigour is involved?
Well, they're not laws of nature or anything. But pragmatically, how could it be otherwise? When it comes to bridges and buildings, correctness beats timeliness, but when it comes to software the opposite is usually true. There are exceptions of course (e.g. medical software)
> Maybe we should reserve the term software engineering for the applications of software development where rigour is involved?
Agree.
If I was burdened by regulation and red tape I can assure you my org would be less secure not more so
If the issue is about teenagers being too immature for the product X, then typically we ban it or age restrict it using legislation (eg tobacco, alcohol, porn, etc)
I think we as a society need to figure out where social media fits in and how we want to contain it and it's still early days for that. As much as I like the idea of an age restriction, I look around the world and see 6 month olds playing with iPads, 6 year olds on tiktok, etc. as the more 'average' way to parent and wonder if I'm in the minority. My kid is 4 and has never touched a device as entertainment, we taught him how to eat out at restaurants/be in public instead of pacifying him. But that seems like an old fashioned parenting method.
Ok, he's technically used it briefly as a viewfinder for the camera and to facetime his grandparents
We've had social networks for twenty five years.
It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible.
A codebase is not encumbered with the physical constaints of an old city. It's not like trying to figure out the physical state of a certain area of London 30 meters deep — which might be impossible without digging and therefore disrupting other structures in the area. A codebase is simply instructions, which are readable and understandable (with the exception of black-box machine-learning models, but those at least have defined inputs and outputs).
If it takes a while it will be out of date by the time it is complete.
A single request might directly impact a dozen separate systems, which in turn call other services. You won't be able to find anyone who knows much past the layer they directly interact with. All of these services are being worked on. Some older ones are in the process of being deprecated in favor of newer ones. Someone is already working on the design of the replacements for the newer ones. Changes are being pushed weekly if not daily. If you send a request and then send it again a minute later, it might go through a completely different set of services based on a feature toggle.
You can't step into the same river twice.
You might have two independent datasets that can't be stored together due to privacy or security regulations. The teams who work on that know and understand that. But some other team working on something completely unrelated might join those datasets for some non-nefarious purpose, which then creates a new dataset with those data combined. Now other teams who also don't know about the regulations might build on top of that combined dataset. They might even generate new data fields from the data that's not supposed to be joined. This stuff can propogate through layers of systems, queries, service calls, analytical engines, offline spreadsheets, etc. It's very difficult to keep track while keeping your engineering teams independent and nimble.
It's worse than you think. First, a little background: I worked on one of Facebook's largest storage systems for two years. So let's talk about some concrete examples.
I knew the rebalancing code better than anyone by the time I left. This is a pretty essential bit of functionality, running regularly on any cluster that has been up for any amount of time and moving quite a bit of data each time. After a chunk was copied, my code would go to delete the old one. I know for a fact that the deletion could fail without my code even getting a useful error from lower layers. I found and fixed many such cases, but I'm sure more remained. That could leave an "orphan" chunk where nobody would know to look for it, and chunks were self-identifying enough that if enough such orphans existed they could be reconstructed into a whole block possibly containing user data.
I knew the data-repair code almost as well. Same problem. I knew the system-repair code less well. Similar there too. In fact I was involved in pulling back "repaired" systems and using them to recover data that would have been lost, more than once. I watched other engineers get rewarded for cleaning up petabytes' worth of no-longer-reachable data (because of cases like I'd mentioned, or bugs, or whatever) that was still taking up space on our millions of disks - again, multiple times. And that's all just one storage system. I'll bet others had similar issues. Also, the problems with truly erasing disks and particularly SSDs are pretty well known. If a machine was taken out of our system entirely and repurposed for another one, or vice versa, there could still be data on its platters/chips that could be recovered with sufficient forensic effort. (These things were physically destroyed before leaving FB, and I've even seen the impressive machines that do it, but not between "lifetimes" within the company.)
So knowing the current code is not enough. You'd have to know every past state of the code during a relevant timeframe, including what bugs it had, which is challenging to say the least. Every configuration detail, too. You'd have to know every rebalancing, reconstruction, or repair event that might have affected each disk. It really is like figuring out the physical state of an area in London. Nothing short of scanning every sector, even as more hardware enters and leaves the system almost every minute, could produce an absolute guarantee.
I'm not saying it can't be done in any system of similar size and complexity. Just that it couldn't in Facebook's, even without malice, largely because so much code was written without that in mind and it's a really hard thing to bolt on afterward. I know there are people working toward it. It's just going to take longer than most people - even technical people - think, and saying it's done would be a lie.
Meta is choosing not to know or pretending not to know. They have the resources to know where your data goes.
No such law exists for Facebook, so they don't care. Someone needs to pass a law with a big fine attached and then things will change.
It's like when in an interview Buffets says it's very hard to know where all the money is in the financial system.
Sure, it's a complex system, but it's more of a matter of incentive.
Full data lineage is a complex task.
A very simple example would just be that a database is destructively updated by some script at some time. What's the log here? What ran? Can you actually replicate it? Do you have full backup of the data before?
It's so easy to lose this.
In the same way you can't enumerate the atoms or molecules in a human body. sure, you can come up with good estimates and summaries that are useful, but not the actual full and truthful state of affairs.
In addition, you are a self-referential participant in the system: so acting in an attempt to observe such likely immediately makes your information at least partially incorrect.
Additionally, if we can't get full/transparency or introspection into machine learning models or neural networks (and we can't because i don't believe we even possess the theoretical knowledge of how to do so), it should also be wilfully apparent that we can't actually know the true lineage or dependencies of systems that implement such in production.
Not to mention the difficulties as the inputs/outputs of such pass in and out of the notional entity or system being observed.
Processes to access personal data vary wildly among tech companies. At Apple, when a machine learning engineer wants to store and use data on the server side, they need to go through layers of approval from lawyers. Even when approved, it often comes with serious constraints about what can be done with the data. Meta is a lot more lax.
A company like Meta that has been building and advancing the field of ML, AI, and even falling foul of the ethics/morality of large scale public manipulation and political campaigning has already demonstrated they have the wherewithal to find where their data exists. Period.
They are just poor stewards.
It's easy to delete old data that is never accessed, so I don't get your point...
Why is this OK?
Most companies found a way do do it. It was a long and often very painful process, involving everything from data entry to backup processes and procedures, but somehow we managed to do it.
Government contracted construction workers: We Have No Idea Where Your Tax Money Goes
When tasked with answering the simple question "Which specific bricks did my tax money buy this year", the two veteran construction workers looked confused and tried to explain that that's not really how taxes work.
The special master at times seemed in disbelief, as when he questioned the engineers over whether any invoices existed for a particular road building contract. “Someone must have a receipt that says this is who the money came from that bought these bricks”
I'm not sure the analogy fits 100%, but it's the closest I could think of. This reads like the author thinks the Facebook algorithm is a human-readable decision tree that only takes into account the data of a single user at a time.
As usual, the problem is not data "collection" or "retention" or "privacy", but "creation". Regulation will always remain woefully inadequate to control such organizations, and the only solution is to adopt systems that don't spew user data everywhere in the first place.
FB uses the data it can’t find or tell you anything about to successfully sell ad space to third parties targeted back at you.
Let’s make it concrete.
We know FB keeps track of which webpages you visit. We also know they use that data as a way to help target ads. That data isn’t in the data export they give you, as far as I can tell from a brief search.
Is there data coming in about which Instagram profiles you follow and what pics you clicked and liked? Is there other data they keep track of? Undoubtedly.
I’m not surprised nobody knows the answer to what all it might be. But pretending that data is fungible like money is just misdirection.
The hearing seems to want to get more to the heart of the issue, but is inept to do so. The stored data isn't the concern, it's the predictive capability of models built on it. The decision of a predictive model isn't any more traceable to a single piece of data than a brick in a road is traceable to a given taxpayer, which is what I was trying to get at with the analogy.
After reading some of the transcript, the special masters seem very on the ball, and it's more the spin of the article I take issue with.
It takes all of Bob's activities and pushes them into bins based on how they characterize Bob; Male, lives in Iowa, 18-25, etc. a bin (Moves to LA, for example), his data will contribute to different bins. This activity is disconnected from Bob at this point; and the data is aggregated away from single interaction events. "Bob visited foo.com" as a single event is gone at this point.
The models grind on these aggregate data bins.
Then when ads are targeted at Males who live in Iowa, aged 18-25 -- those ads get shown to Bob, because he is tagged with those tags.
They don't "keep track of which webpages you visit", not for more than a day. Those events get pushed in large aggregate stores of activities pretty fast. These aggregate stores are vastly smaller than if you kept all the individual data, hence much cheaper.
I think it would be a more meaningful analogy if you had posited two government accountants.
Nobody could possibly manage Facebook as a centralized single entity, but it's hard to imagine it any other way from the outside.
Now, sometimes systems diverge from the design, and sometimes threat models are incomplete. But the exercise of generating a threat model makes understanding who manages data more manageable.
There easy! But also useless! The data is everywhere. That's why every "where" was created. Everything is a potential threat vector and needs to be protected by layers of fallible security.
They don't even know how to ask the right questions to get to what they want to know.
Context: I worked at Facebook
It's 850 pages long, and the few spots I jumped to the guy kept answering with "I don't know", "I'm not familiar with this", etc. Even the parts he says "I'm familiar with this service", the guy goes on to answer "I don't know more than what's written there". The guy couldn't even answer if certain databases were accessible by third parties (not implying they were).
This must have been painful to be a part of. 100 variations of "I can neither confirm nor deny".
The where part is also alluded to as to mean "systems" from what I gathered.
They need to have an exhaustive list of how they're using private data, and they need to have a process ensuring that their engineers are not adding new sources of private data or not using existing private data without the company approving and updating that list. Yes, their current processes aren't fit for that - as Meta documents quoted in TFA say "We do not have an adequate level of control and explainability over how our systems use data, and thus we can’t confidently make controlled policy changes or external commitments such as ‘we will not use X data for Y purpose,’" - so these processes must be changed.
If you’re Facebook you have to know this in order to process that data. You can’t pretend not to understand the question.
You're right. If the questions were being asked of someone who was genuinely motivated to answer them, this wouldn't be an obstacle.
The phrase "all your personal data" (or the quote from the article, "every bit of data associated with a given user account") feels vague and imprecise only if you have an internal understanding that for practical purposes, it really is impossible. The Facebook engineers know that and everyone here knows that. It's not literally impossible - an internal team of digital archaeologists with unlimited resources and no deadline might be able to do it for a single user. But practically impossible across the whole userbase.
If you were genuinely interested in providing insight to the hearing, you could explain all that, and then explain how you could go about creating a rough sketch of an answer, and how you'd go about adding some detail to that sketch.
The Facebook engineers have no real motivation to provide insight off their own bat. They can take the question literally, and answer honestly that they don't know, that nobody else knows, and that it's probably impossible TO know. I'm not even sure if this is acting in bad faith. I can easily imagine feeling cagey in this situation, and responding as such. Experiencing an emotion isn't acting in bad faith.
All that said...these questions were asked by a court-appointed subject matter expert. Either the subject matter expert is not the right person for the job, or...and now I realise that of course I haven't read the transcript and there's every chance that what's quoted in the article is not the most insight the court managed to prise out of the engineers. posting anyway in 3 2 1
>What does it mean "all your personal data"? It means all your personal data.
>What does "where" mean? Where facebook stores it.
This is only a complicated question if you want to start saying, "Well this isnt really their personal data, we just use it for x..." If its data related to my account or me as a user it's my personal data.
Maybe its difficult to track all that down but certainly that is a challenge worth tackling in a world where GDPR exists.
For example if you asked me now if I know where all the receipts are for my tax-deductible donations, I would say no, I don’t know. Why? There is no reason for me to know right now.
But if the IRS told me I was being audited and must produce those records or go to jail, I would find them. And I have confidence that I could, because I know that in general, those are the sorts of records I keep (somewhere).
Most Facebook employees do not need to know “where personal data is” to operate their business, so they don’t know that. But at the same time, Facebook could not operate if the data was “nowhere” (destroyed), or if it was randomly distributed with no queryable structure.
So the question is not really what they know now, it is: what incentive do they have to go find out.
The amount of data that we all see, every day, that is grossly mishandled may signal the end of the ML and AI goldrush. You can only build models, and run data through models, with the consent of the owner of that data. Large producers of data (think vehicle fleet operators) are beginning to take ownership of their data, and are only _licensing_ it to processors for very specific purposes. In the example of vehicle fleet operators, they may only want route planning, and not have their data used to sell them tyres based on mileage. Also, while governments may be busy with other stuff currently, at some point they may decide to turn on the regulatory screws.
Oh you sweet summer child.
I have considered pivoting my service into a privacy respecting data store ( https://www.adama-platform.com/ ), but I've yet to meet anyone that actually cares about user privacy to rethink their data layer.
Of course senior engineers don't know where the data comes from exactly. Does your mechanic know where each tire comes from or care? Does your fav restaurant chef know each field that a produce came from? The suppliers know that info.
I know beyond any doubt that there are most definitively some people that know at least roughly where most data about most subjects are at Meta. I'm sure the data is a vastly beyond what a human can process. However some people there know where to go looking. The truth is in the Piles of money they generate by selling that data.
Awww gee wizz your honor. We have no idea how we do it. People just keep throwing piles of money at us so we keep taking it. I don't know why they do it. We definitely don't have mountains of highly specialized data on our users that we tell them we can get exactly what they want then provide that info to them. Somehow though when the court asks without piles of money we just can't make that magic happen...........
"At least roughly" is doing a lot of work here. I think what you're getting at is that there are people who know which storage systems are likely to hold that data, but knowing exactly which objects do is far more challenging. Copies get made while data pipelines are tweaked, and then forgotten. Copies get made internally on storage systems as disks/machines fail and get repaired, as rebalancing occurs, and in other cases that typically only one or two members of the team understand at all. (I worked on one of Facebook's largest storage systems for two years BTW.) Copies get made for AI training, experiments, shadow-traffic generation, etc.
Could they do better? Should they do better? Yes and yes. But it's not a lie to say nobody knows how to find every last orphan and fragment. They can pretty reliably delete everything they could find to sell, but that's a very different thing and you're conflating the two.
I'm quite certain that if I showed up to FB with a $50 million dollar ad campaign to run they would have quite the opposite story about their data knowledge. Marching dozens of people into the room promising they can slice and dice every bit of info down to the atom for my needs.
Yet in court they want to claim they live by the seat of their pants. Its a deception tactic. Even though I'm sure its as you describe partly true in technical concept.
I do believe that FB is the least privacy invasive of all the big tech giants for what its worth. Nothing against them. I actually think this case is an intentional aggravation to the courts to have regulation created on data collection. That way FAANGM can cement themselves as the offical national data brokers with costly startup regulation for any compeditors.
Based on what?
> they would have quite the opposite story about their data knowledge
As I said, they can probably delete everything they could sell. What's left are fragments, which are still a real concern but not enough to use for ad targeting and certainly not enough to satisfy a big-money customer. They might take that $50M anyway, but it would be under false pretenses and that's a very different issue.
“I don’t know,” answered Zarashaw. “It’s a rather difficult conundrum.”
This is not a basic question because of how ML works. Can we say a system contains my data if my data was used as a training input to a neural network at some point in the past?
In that case, can I sue anyone using Stable Diffusion for stealing my data because the billions of images in its training set included something I created?
No, because of the landmark Authors Guild, Inc. v. Google, Inc. case which found that ML training can be considered fair use in some circumstances.
Copyright law covers things that are expected to be published - i.e. creative works. We decide that authors get to control how and when they are published. Privacy law is different: the subject is information that is held in confidence and should not be published under any circumstance.
Fair use would circumvent the "copyright my nudes so I can DMCA them off Facebook" kind of action, but not, say, the GDPR. GDPR does not care about fair use, and its skeptical of TOS agreements.
I can't explain somehow this is the average candidate, yet somehow... Life goes on.
(1) Is it possible to be sure that the primary copies and backups are gone, so that finding anything that's left would require some very specialized knowledge and/or an infeasibly massive scan? I believe the answer to this is probably yes at this point.
(2) Is it possible to be sure that absolutely every last vestige of the person's time on Facebook is gone? I believe the answer to this is still probably no, and likely to remain so for some time. At the very last, some artifacts will remain in those opaque AI models.
I suspect the same two answers exist at many companies. What is HN's data retention policy? Oh, oops, none of my data would be deleted if I left. I suspect that Google and/or Apple, maybe Microsoft as well, are a lot closer to "yes" on that second question, but even then I suspect gaps appear from time to time.
I say this not to condemn or defend anyone, and I know companies under stricter regulatory regimes can give more definite answers. It's just the state of the art at Big Tech companies as I understand it.
Well in order for your grandma to log into Facebook her user account must have a primary key associated with it so she sees her info when she logs in and not someone else’s.
We are talking about computers and databases. When did using a computer to search a database become a difficult, nigh impossible thing to do?
Even if design documents and flow charts or whatever don’t exist could they not fairly straightforwardly be reverse engineered by taking a sample of users and searching all databases for associated information?
This seems like a transparent ploy on the part of Facebook to avoid regulation by casting the perfectly doable, searching a database, into some sort of incomprehensible impossible task. The credulous author of the article and many comments here seem to be strangely buying into it.
FB investors don’t seem to have lost faith in the company’s ability to search databases. When it comes to making the company money from those searches, no problem at all. But regarding lawsuits or potential regulation, we can only throw our hands into the air and simply wonder and gasp at this incomprehensible mess we have made.
“We do not have an adequate level of control and explainability over how our systems use data, and thus we can’t confidently make controlled policy changes or external commitments such as ‘we will not use X data for Y purpose,’” the 2021 document read.
Or that one time I found a dialup modem under the raised floor and attached to the Equifax (then TRW) mainframe.
Or catching someone inserting a USB stick into a PC inside a secured white-lab area.
Absolutely crazy times for large entities.
I guess storage is cheap enough, but even for Facebook, being able to save say 10% on storage must mean something.
The cynic in me wants to say that not documenting certain decisions is a common method of hiding immoral decisions.
What ends up happening is each part of the system has a "point of contact" who is the go to for that piece. Lots of decision history is in the heads of team elders. Hopefully they are still around!
In my experience there’s a ~10% chance an up to date piece of easily discoverable documentation exists for any service or component, and there’s a ~90% chance at least one out of date document exists. The later seems much more harmful, like when you realize some PM has been referencing an out of date doc for the last 9 months.
I mean people "usually" wont say that out loud of course.
>We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.”