Government agencies buy commercial data about Americans in bulk | Hacker News Reader

Government agencies buy commercial data about Americans in bulk

npr.org

273 pointsnuke-web381 comments

Netzpolitik.org actually reported on what you can do with this type of data a while ago. They tricked a databroker into getting a free sample of geolocation data, 3.6 billion datapoints. They were able to build individual movement profiles for people and link that with real identities by putting just a little bit of work in. For a government with access to stuff like palantir this would mean a full movement profile for pretty much everyone with a phone. German article about movement profiles: https://netzpolitik.org/2024/databroker-files-firma-verschle... Broader article about their research into the databroker topic: https://netzpolitik.org/2024/databroker-files-die-grosse-dat... Wired article for English speakers: https://archive.ph/DmWrw Wired frames this a little strange, around how the government is powerless to stop it and such, especially considering how they now actively admit this is in their interest.

There was a great piece published back during the Patriot Act debates where a princeton or harvard professor used modern math techniques and tavern records to triangulate for arrest the early Patriots and their meeting spots. It was a great article.

Got a link to this? Sounds like a fun read.

Not OP, but here you go.

OP had it slightly wrong though: it's not tavern records but membership lists of colonial Boston organizations, and the author is a sociology professor (Kieran Healy), not from Princeton or Harvard.

He uses basic social network analysis on historical membership data to identify Paul Revere as the key figure among 254 colonists using nothing but "metadata." The whole thing is written as a satirical report by a British intelligence analyst in the 1770s, which makes it a pretty effective commentary on the "it's just metadata" argument from the NSA debates.

Link: https://kieranhealy.org/blog/archives/2013/06/09/using-metad...

Isn't he also making the point that's a very effective way to triangulate the leaders of an org? That's just going to reinforce the NSA's inclination to do so.

I think the argument is that NSA already knows exactly how valuable metadata is, while the average person significantly underestimates its importance without a concrete demonstration.

The same NSA that publicly states "We kill people over metadata"?

If that's the type of things they say publicly at conferences we can only imagine what a more sensitive comment would look like. How anyone can underestimate the importance of that is beyond me.

spread a false story in public news and monitor cell call graphs.

I remember some journalists used (currently legal) meta-data from data brokers to track the movement of some politicians and later confronted them with it: they were now very much opposed to this being legal.

Now, it seems like someone would need to do that for capital hill .. and then make sure politicians are not voting a law that only exempts them from meta data collection and usage.

Someone really should put up a website that tracks US politicians' locations in real time. It's clear that our government only cares when directly and personally threatened.

They’ll just make a list of which people it’s illegal to track, and keep tracking the rest of us.

Not only is it likely very easy/possible to track everyone with a phone but it has now got to the point where "movement without a cellphone present" is a red flag.

"Hey...why is this guy suddenly deviating from his normal routine? License plate readers show him 100 miles out of his normal area. Why did he leave his phone at home?"

Just like social media. Not participating is considered suspicious.

Anyone with Govt. level access (or billionaire level access) can very easily put all this data together.

The state of the art has advanced so far in doing this. I remember way back in 2017, 9 years ago now, at the Scaled ML conference, Claudia Perlich gave a presentation about using RTB data to target ads. When she got to slide 23 [1] my jaw hit the floor. This was a small ad targeting company, and again, 9 years ago. Here's what they publicly said they had:

Consumer Events:

• 100B DailyEvents

• 20+ data integrations

• Clickstream

• App usage

• Ecommerce sales

• Cash register sales

• Precise Location

Context Data:

• User

• Device

• Location

• URL

• IP

• 200 Million Devices Daily

Universal DataStore

• 50 Trillion Record Consumer History

That's about 150,000 datapoints on everyone in the U.S. For a small company. In 2017.

[1] https://cdn2.hubspot.net/hubfs/6212008/ScaledML%20Media%20Ar...

The personalization stuff is why I avoided ML like the plague - all these folks making huge money but all of it to build a surveillance state for advertisers. Already having value my privacy enough to never work for an ad revenue company, they all seemed beyond the pale.

See also Dark Data from DEFCON 25: https://youtube.com/watch?v=yWqdTVQsnPg

But it doesn't need to be marketed in such a sinister fashion. In 2012 when Google Maps informed me of delays along my usual commute, complete with a GPS trace of my route home, completely unprompted, I started turning off location history (lol, yeah right). I didn't even know they were collecting that data, much less analysing it that hard.

These days, that would be considered a feature - not a dystopian hellhole, and you would be a Luddite for turning off this new smartphone augmented brain. The product will make you happy. [0]

Welcome, to City 17. You have chosen, or been chosen, to relocate to one of our finest remaining urban centers. It's safer here.

[0] https://youtube.com/watch?v=5PZ73nLZaqc

Buying commercially available location records from data brokers would be far less concerning without the capability to, per Anthropic’s CEO words, assemble from that data “a comprehensive picture of any person's life—automatically and at massive scale”. It’s a world of difference between when you have to work hard to construct (and keep up-to-date) such a picture for a single individual, and when someone can do it for an entire city with no effort.

Exactly this. I don't that believe most people can grasp the scale, speed and complexity at which this is has become our reality.

And, unfortunately, this is likely a major (potential) revenue source for AI companies that are all struggling for cash.

Sure but the possession itself of that data without a warrant violates the spirit of the 4th ammendment. So its time that loophole was closed so its not a an issue anyway

I actually would be fine with the authorities having the ability to process this data to solve crime and stuff, but only as long as there were checks and balances and it was happening according to the constitution, which it is not right now

> possession itself of that data without a warrant violates the spirit of the 4th ammendment

Does it? An 18th-century tavern owner could keep tabs on the comings and goings of their customers. It would have just prompted pushback when they started sharing that list.

Possession isn’t the problem. Sharing is.

Really?

No, the bar owner has a right know who's in his bar.

The local or federal government do not have the right, or need to know the whereabouts of the average law abiding citizen. There is no "free" information, all information has a cost, whether it be acquisition or storage. Currently the people are taxed to oppress themselves. There is no choice not to be taxed, there is no consent.

You've said shat I meant to say more eloquently. That's my point. The bar owner can possess the information about their own bar without problem. The issue is when they share it with others, voluntarily or under coercion.

No you still don't get it, the bar owner can share the information with whoever he likes.

The government, does not own the country, it's not "their bar".

> The government, does not own the country, it's not "their bar"

Never mentioned the government. If I shop at a store and the store owner starts selling my information to everyone under the sun, I'm going to keep an eye out for alternatives. I don't mind them collating it. I do when they share it.

I feel like the example you want is a Video store owner could store a list of what movies congressmen rented.

Which was trivially not covered by the 4th amendment [1].

[1]: https://en.wikipedia.org/wiki/Video_Privacy_Protection_Act

Government is that institution in our society which possesses a monopoly on violence and should be held to a higher standard than a tavern keeper.

In this context, I believe the tavern is a metaphor for Facebook etc., and hence it's not one tavern but a business which tried to own all taverns, pubs, and restaurants, who has made the beer (and food) free because juicy gossip sells more opportunities for ad revenue, and all the governments want in on that.

Also, "monopoly on violence" is deputised in a lot of ways, including "Stand Your Ground" laws, and "Castle doctrine" (which may or may not include a workplace), and what's allowed for trespass and if trespass includes not leaving when told to.

(And even when it's more of a first amendment issue than a fourth, there's also occasional news stories about people getting sued for leaving negative reviews of a business because the business snuck in a no-disparagement clause into the terms and conditions).

There is a big difference between a tavern owner keeping tabs on the comings and goings of their customers and the government having 24/7 precise location monitoring on everyone in the entire country.

One does not violate the 4th and the other does (though they do it anyway).

I think that comment specifically meant possession by the government.

The tavern owner is not the government. The bill of rights is about restricting the powers of the government, not of tavern owners.

I expected years ago that the government, at some point, would realize if they are interested in the data that they could purchase, other nation states would be as well and could use it against us. Therefore the logical conclusion would be to declare collection and sale of such data to be a matter of national security and strongly restrict it as such.

The detail I failed to understand at the time was just how much money there is in data collection and brokerage.

The other detail you missed was, that this world is mostly not run by sane governments that do the rational thing you would expect from them.

The irony of npr.org welcoming me with

"We and our 474 partners store and access personal data, like browsing data or unique identifiers, on your device."

474!!!

Not ironic, unfortunate reality. I'm annoyed npr and 474 use personal data for profit. It should be illegal. I'm terrified the government uses personal data without checks on the use case. Especially since it is illegal, and they just don't care about that. Or there's a loophole, or whatever, and they just don't care how The People will react to them using it, which is just as terrifying.

Wait, is 474 a number or a proper noun?

Whoops, I assumed it was a proper noun the way I read it. Like a company that does data collection or something. Whatever, I'm not gonna bother to look at it again, I'm quitting the internet.

Some of those "partners" may be storing your precise geolocation for 12 years: https://x.com/dmitriid/status/1817122117093056541

Not enough people are talking about this. It seems to me like the vast majority of people just don't care, primarily because they don't understand the ways it could dramatically impact them in the future. Short term thinking is a scary phenomenon.

"I don't have anything to hide," sort of foolishness.

This statement is from those who don't understand what's possibly at stake and those who are afraid to find out. Who can say how bad this will be for the kids in high school now, in the future? (Or most likely sooner.)

While we in the US argue about politics, we ignore what's happening behind the curtain.

For a deep dive into the topic, I will recommend the book - Means of control by Byron Tau [1]

[1] https://www.penguinrandomhouse.com/books/706321/means-of-con...

FBI director was asked point blank if he'd commit to not buying Americans' location data. he said no.

This story cones uo time and time again, people rail about the data buyer, but practically speaking any one or thing can buy that data and use it against you and yours. The very collection/assembly of life data is dangerous.

Can't we just buy the data of politicians to make them aware of the problem?

Listen, this is nothing new. You can find articles like that going back years and years. The truth is: convenience trumps privacy in practice in a lot of cases. Two examples:

1. Theoretically speaking, my (data) privacy is of a high value to me! -- Then you should stop using a smart phone. -- Well...

2. I don't want anyone to create a profile of my habits because it's none of their business! -- Hi, do you have a Walmart+ card? -- Sure, here you go!

And I actually like the concept of reward cards (although I don't use them) because it is pretty much the only way how you can make money off your data.

> Listen, this is nothing new.

"Violations of your constitutional rights have been going on for decades now, so it's time to shut up about them" is certainly a take.

Action speaks louder than words. It doesn't matter what people say they prefer, their actions reveal a true preference.

This is just idiotic to say. Nobody actually prefers to have their data siphoned off, in the abstract. If you make it a choice between "privacy" and "being able to participate in society in any way at all", obviously they're going to pick the latter. That's not a "revealed preference", that's coercion. It doesn't actually have to be that way. We can have a world where we have smartphones, and the government can't use those smartphones to track your location at all times.

Funny how we stopped drinking lead when they stopped putting lead in our water.

If you want to absolve yourself of personal responsibility, go ahead and blame "them" and "we".

You were free to drink lead-free water the entire time, you just didn't care enough to do so.

We also stopped breathing fall out when they stopped doing atmospheric nuclear tests.

If only those lazy 1950s layabouts carries oxygen tanks instead of complaining about cancers.

Let me know how I can participate in society without a cellphone, credit cards, sharing an ID with businesses, showing my face to ubiquitous cameras, whatever else. I'll do it. Tell me how.

Putting words into other people's mouth isn't the best etiquette: where did I say "it's time to shut up about them"??

Your tone implies it. "Listen, this is nothing new" is a phrase dripping with "I'm tired of hearing about this". You surely know that the people pointing out continually escalating violations know that the violations are not new.

You inferred it. They did not imply it.

They might simply be tired of listening to armchair protestors who don’t take even the most basic actions to backup their words.

I read the implication too, as well as the fatigue.

They offered nothing to counteract the idea that we should just shut up and accept it. Then they closed with "And I actually like the concept of reward cards (although I don't use them) because it is pretty much the only way how you can make money off your data." - which sounds like they have given up opposition, and are now considering ways to profit from the situation rather than fight it.

Then don't be in the thread.

The majority of the population has been ok with this path for a very long time so it’s unlikely to change.

There are basic ways to act, not just talk, to support resistance to this path. And people, even some people reading this very comment, are unwilling to take those basic actions while also whining loudly and/or downvoting in angst.

There is nothing "basic" about preserving your privacy in this age. I go to ridiculously great lengths to preserve my privacy. That entails using VMs with separate VPNs for every different thing I do on the internet to avoid cross-pollination between my online identities, that entails never taking my smartphone out of the house, that entails using burner phones, that entails accepting that I simply can't use an increasingly large number of services that are being gated by identity verification, which is now trying to be forced on being able to use a computer at all at the OS-level. It is an absolute pain in the ass to worry about this, and it's completely understandable why people give up, but that doesn't mean they actually want it to be this way. Privacy should be the default, not something you have to fight for.

It's a direct violation of the fourth amendment. The worst thing you can do is just accept it, as that normalizes it. This is an end-around to avoid going through judicial channels to obtain information about private citizens, full stop. I'd love to hear about such brazen examples in the past, as right now, we have Kash Patel openly admitting to this activity either out of ignorance or hubris, either of which is terrible.

>convenience trumps privacy in practice in a lot of cases. T

I quite recently found that the "CRED" app works by requesting permission to access the user's WHOLE INBOX and reading their mails. The users apparently have no problems giving access to their all mails to some app that they don't have control or ability to scrutinize.

Even some package tracking apps want access to your email to scan for packages.

Forget that!

Reward cards aren’t for your benefit.

It is betrayal.

Governments betray the people. That is by definition betrayal.

In this context "age verification" must be seen as an extension of betrayal. Why does a government want to sniff after user data suddenly and make it a law?

I heard this on the radio this morning. The NPR story reported the danger as individual’s PII can be combined through the power of AI.

I remarked how curious it is that wondrous AI should be the technology people need to experience before they can imagine the dangers of Data Brokers and the Mother Of All Databases (MOAD).

Which is why Mazda torments me and disables my ability to remote start my car without "connected services" app installed on my phone.

I've said this ad nauseam, but their TOS says they sell to 3rd parties including law enforcement and insurance companies.

Nobody cares though. Main point. Like 80% of the public have no idea or interest knowing what connected services does. If you tell them. They still don't care (my parents).

I remember term "privacy laundering" and "surveillance capitalism".

Example 2019 article https://www.lawfareblog.com/facebook-encryption-and-dangers-...

In reality nothing new.

Do we have a list of who is doing the selling? (Directly and indirectly.)

People talk about this as if Snowden and the NSA stuff way back meant nothing. Why the hell would they need to BUY your data? They're the US Government ffs. They have, and they will, demand that data from companies regardless of financial compensation. These articles don't seem to convey that very well.

Hot take: It's about to become a lot more common and as strange as it may sound, I don't blame governments nor corporations - the people are to blame. Everyone with a functioning brain saw that coming decades ago: If the long haired 12 year old punk, that I was in the early 2000s, could see a problem, chances are, you don't need to have 20 PhDs to figure it out either. But most people ignored it and carried on sharing every single thing about them 24/7. I have a friend who is furious about it now and when I call him out, being constantly on facebook/instagram/whatever, he still refuses to accept that it's his fault. His argument is that he needs it cause he has a music band. Here's the kicker: he's had multiple over the nearly 2 decades that we've known each other. And the reality is, he's never made any living or money out of it and acknowledges that he never will and he's only doing it for fun while investing his own money into what is effectively a hobby. He's made a couple of hundred bucks at most and much of those have been from close friends of his as a form of support and personally in my case purely out of support since the music he makes is not my cup of tea. And yet, every time he farts, he posts it on some social media. "I'm sure we are not far being tracked on the roads and if we exceed the speed limit by 2km, we get a fine instantly". I'm genuinely starting to think that people are not capable of making the connection between cause and effect. Best case scenario, they see those as a coincidence. A few weeks ago I logged into linkedin from an anonymous account and I was horrified: random posts from people I've never heard of: "We created the best performing AI that can do {something}. Comment on this post and we will give you an early access to our product."

And dozens of comments underneath, as you might have thought. The company - one 19 year old kid running a node frontend on vercel. First comment - from a marketing manager at some FAANG. If this is not a sign that we are cooked, I don't know what is.

People keep sharing everything they do online, rely on cloud based llms which clearly collect their information. And everyone and their dog understands that AI companies operate at huge losses and promises they will never be able to fulfill. Sooner or later the investors will start asking questions. Governments are in this bizarre place where they are part of this on two fronts. At large because governments are lead by people in their 60's and 70's and have no goddamn clue what AI is beyond magic that can do anything (or so they are made to believe). So they are pouring money into AI companies to do some ridiculous tasks for them, while also pouring money into collecting data. To their minds, it's probably "we have the data and we have access to the all-seeing and all-knowing ai". And while that is happening, sloppers ask that same AI to write their code, where to buy dinner, use it as a therapist, relationship consultant and all that, adding more highly personal data into the bag of data that should remain personal. Forget how bad corporations have been at preventing data leaks. When the investors start knocking on the door, asking for their money and a government asking for a JSON containing your medical records, private information and whatnot, guess who won't think twice about it and happily take the briefcase full of cash.

Ultimately, Idiocracy was supposed to be a comedy, not a documentary but here we are.

These social media platforms have some public information which can be within easier database access to be sold for the highest bidder but the context of this article is about all the other data like geographical data, Ip location and which phone etc tracking which is also tracked by these social medias and sold to the highest bidder in this case the govt

Now There is point saying that we should use better alternative forms of social medias like mastodon etc. perhaps hackernews and that can be a worthy discussion but I have thought about it and I do feel like your musician friend is right in the sense that it might require some presence in social medias for some purposes.

Thinking about it, one of the largest pieces of advice I feel like is getting converged is that the best place to become entrepreneur is being in the space where you might sell your product. So if I wish to sell tech related products, I am fine with only using hackernews for the most part.

In a similar fashion to that, to gain visibility, These musician go to these platforms and many do hit and many don't and sometimes its a matter of both hard work and luck.

Now that being said, every message that you wrote about your friend feels a bit bad-mouthed.

Most musicians barely make money from music. Most artists don't make money from art - and I should know, being the son of two. You are either the top seller/musician or you are everyone else and in 99.99% of all cases, you are everyone else. Here's another example: a friend of my mom spent the last decade pushing her paintings on etsy. Then trump added all his tariffs and she went from making a minimum wage or there abouts to a literal 0. Meanwhile my mom never bothered with any of those and simply gave her paintings to two local art galleries - no websites, no social media, just tourists walking about and even if this isn't her main revenue income, she outsells her friend easily. And that is far more common than you think.

Admittedly, I don't think HN is a good place to promote your product either. It used to be a place where innovation and doing something complex was appreciated. These days it's all about people praising slop.

As for my friend - I've said it to his face multiple times but as they say, you can lead a horse to water, but you can't make it drink.

> Admittedly, I don't think HN is a good place to promote your product either. It used to be a place where innovation and doing something complex was appreciated. These days it's all about people praising slop.

I do sort of agree with that, I mean I literally saw within another thread related to music (Misfits) where this guy https://news.ycombinator.com/user?id=hustleracer is clearly using AI and just joined

I am seeing lots of people use AI within hackernews now. I have flagged them for what its worth but yes, I do sort of agree with that hackernews has decreased a little bit in value.

But I used to hear the same thing a year ago as well, I would still be considered relatively new to the forum but I heard stories of how one day PG would just decide about elixir and have front page all about elixir/erlang and people sometime ago reminisced that, which happened many years ago.

But even with all of this, I feel like hackernews is a place where sanity is still intact for the most part and there is still some authenticity more than other places but that's just my 2 cents.

I do agree that musicians struggle with making money sometimes. Its definitely a winner takes all market from my viewpoint and median musician doesn't make much but the mean is skewed because of the billions racked in by famous people.

I am not exactly sure how to preserve Music,Art though. One of my closest friend said to me that her sister wasn't studying well and now my friend is 99 percentile kind of fellow, but to me that moment, Man it felt like that poor girl was put into expectation by her brother and her family and sometimes feeling too. So I said to him that hey if she wants to pursue music/art/anything, then let her do that and my friend basically told me again about the struggling economy of that.

I am not sure what can be done with all of this. Universal Basic Income seems to be the solution. I think Ireland passed UBI for artists sometime ago, maybe that might help preserving music/art.

An answer I feel like is happening is that atleast for my generation, it feels like a lack of culture. I am not quite satisfied by how the social algorithms can promote brain rot but not show music and just, like, I feel like our generation and next generations to come have lost something more meaningful for these algorithms to catch the money trail and the people to make such brain-rot.

My point is, Music/Art has some incredible contribution to the society but society doesn't reward them enough or fairly and then we have the other part of society turning attention into a commodity and churning out content like a factory. All in all, feels like a cultural degradation as time passes from one generation to another.

I'm with you here as well. Not so much on the UBI side though. I only believe in social measures to a certain degree because it is a slippery slope. At the moment, my job is very demanding and sucks up a lot of my energy. In addition the last decade was an absolute bombardment with family problems and it all fell on my shoulders and I was not betting on myself to sort things out at all(even though I managed against all odds and at the cost of a lot of compromises with myself). And 10 years is a long time, especially in the 25-6 to 35-6 range.

Now I don't have all that much on my shoulders anymore or rather it's very much under control but once everything is truly sorted, I have thought about it many times: I am truly exhausted and on a personal level a less demanding and less busy job does sound appealing in a way even for less money. And this is the catch: not everyone is greedy and many people are capable of saying "you know what, I have enough, let's take it easy". Which would become a huge problem on a large scale when the balance shifts. You have over 3 generations now (alpha, z, millenials and x to a very large degree) who have been bombarded for decades by social media and feel no desire to try or learn something new as opposed to just relying on slop. And it was bad enough as it was even before that - I haven't seen nor do I wish to see a large chunk of my family but to give you an idea, my cousin(at the time around 10 years old) did not know how to eat with a fork and knife or tie his shoes or button up his shirt. Not because of a mental disability but because his parents had a child instead of getting a small dog to take for a walk two times a day.

Imagine when you have tens of millions of those that would gladly scroll through tiktok all day long and rely on UBI without batting an eyelash.

I genuinely don't have a solution but UBI does not sound like it. I agree that for most of history, humanity has been pushed by a very small fraction of individuals but currently I don't see people that are doing it, given that we are currently living in a bullshitter economy: "within the next year we will have {x}".

I don't know of a single instance in humanity where someone has made a big leap, granted that all the basics are provided to them in the same way no one has gotten out of poverty through charity.

If I had to base everything off of my own experience(and those around me), my view is that a certain critical mass of unfortunate circumstances are required in order to get someone to reach their potential.

I am a big personal privacy advocate. However, I'm also well aware of how much crime is happening online and on signals that don't sit clearly in public and private channels. If you worked with this stuff like I do, you'd understand gov't and le needs more regualted and responsible access to this data. At the same time, if you're in Brazil (FE) there is no need for Facebook to comply to your warrant request. So how do you stop romance scams, coordinated trafficking, everything that happens there (BTW META is a HUGE platform for this stuff) So sometimes, your only way access data is through purchasing it.

So the US federal government will combat romance scams by violating random citizens privacy when it cant stop the president from conducting stock market manipulation scams on twitter? Institutional trust needs to be earned and maintained.

> So the US federal government will combat romance scams by violating random citizens privacy when it cant stop the president from conducting stock market manipulation scams on twitter? Institutional trust needs to be earned and maintained.

Additionally, we expect the government to respect any laws or acts when the current administration seems to be ignoring most of it? Literally why do people even think the government needs to "BUY" data??? They can get it regardless and nobody is really there to stop them until things change.

Reply on news.ycombinator.com