The Fall of Big Data | Hacker News Reader

The Fall of Big Data

argmin.net

35 pointshenridf32 comments

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think.

I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the country.

There is real desperation in this country. That's why Trump is President. People didn't vote for him because they are racist. They voted for him because they worry about putting food on their tables. They worry about being left behind in a world that has fewer and fewer ways for the average Joe to dream about a future. They're angry, rightfully so, and in Trump, they found someone willing to listen.

They sure as hell didn't find it in us. Sure, we've given them lip service. We've looked at them through our rose-tinted Google Glasses, and at best, we've felt a twinge of regret at the way the world is going. Not enough to do anything. For all the efforts to teach girls to code (a problem deeply related to our own talent shortage), how much have we done to figure out what a blue-collar worker in rural Indiana is supposed to do when the sole factory in his town shuts down?

Because let's face it -- we look down on them. We call them hicks and rednecks. We scoff at their lack of high-falutin education. We think they got left behind because they're not good enough. Because they didn't pass the tests we did, didn't have the ambition we do, didn't try as hard as we do. It's a nice thought if you want something to let you sleep at night, but it's not true. Those people aren't worse than us. They're not left behind because they're stupid. They're being left behind because people like us stopped giving a damn a long time ago.

The solutions we did offer were exactly the ones you'd expect from a far-removed elite aristocracy. Teach everyone to code, or throw them money. Neither of which they want. So the people you laughed at, the people you thought yourselves better than, the people you offered little more to than pie in the sky techno-utopian dreams -- is it any surprise they turn around and elect someone you don't even understand? You never understood them anyway.

Yes. This is exactly what happened and what so many liberals (myself included) don't want to hear but have to come to terms with. It's in our best interest to share the prosperity we enjoy with those less fortunate, or we create a breeding ground for anger and ignorance that will come back to truly make our lives hell.

Same thing applies in the middle east where we've bred terrorism from poverty.

This is our collective fault. We can't ignore them anymore. The only way forward is to empathize with our fellow human beings whose suffering is real and whose anger is dangerous.

It's not just poors who voted for trump. So I think calling a vote for trump a vote for survival is a stretch.

>There is real desperation in this country. That's why Trump is President. People didn't vote for him because they are racist. They voted for him because they worry about putting food on their tables. They worry about being left behind in a world that has fewer and fewer ways for the average Joe to dream about a future. They're angry, rightfully so, and in Trump, they found someone willing to listen

How do you square the poverty story above with the fact that trump ran on a fiscally conservative platform, within a fiscally conservative party that promises to reduce the safety net?

I have to assume that many don't want a safety net; they want to be able to be able to support themselves.

Which is fine... but personally, I don't see that much difference between charging a tax that makes my imported goods more expensive (in an effort to subsidize American workers) and, say, just setting up some sort of reverse income tax, or strengthening the Earned Income Credit or what have you so that you directly tax me and directly give those dollars to people who don't have competitive job skills. If anything, I bet the latter would be more 'efficient' in that I'd have to pay fewer dollars to keep those folks fed and housed, just 'cause if you make imports more expensive, sure, more stuff will be manufactured here, but most of that manufacturing is going to be done by robots, so it seems to me you'd mostly be making me pay more for stuff I buy, then turning around and increasing the demand for labor from people with my sorts of skills.

Not everyone believes that the solution to poverty is welfare. I imagine many of his voters believe the solution looks more like protectionism. Which is basically what his plan is.

I'm not claiming they're right. I'm just claiming that you can be a fiscal conservative and still think you have the solution to poverty.

But isn't protectionism a lot like inflation, in that it can stimulate labor demand while making things more expensive? (And is often called a hidden redistributive tax)

My impression is that this party is as much against inflation as it is against handouts, while inflation occupies the same niche as trade protectionism in that it can stimulate job growth without seeming like a handout.

This is the first I've ever heard of protectionism as a form of inflation; I doubt that the average Trump voter would believe what you're saying here. I mean, I don't believe it, and I voted for McMullin.

Well, technically it's a tax, but it doesn't matter to the consumer if it's a tax, especially one this upstream, that makes a commodity more expensive.

This is the narrative currently being pushed, but let's not forget its weaknesses:

- A blue-collar senior in Appalachia is really unlikely to have enough contact with those "liberal college students" to be bothered with their constant references to Judith Butler etc.

- If the issue is with racism or homophobia, I don't see a compromise happening: minority rights aren't subject to popular approval. Democracy isn't two wolves and a sheep deciding on dinner.

- Trump voters are by and large not the poorest. The white guy with missing teeth living in a trailer is the wrong cliche, because he's unlikely to vote at all. These are conservatives, many retired.

- Every time some politician talks about "real America", he's trying to delegitimize liberals. I fail to see how that is less insulting.

- People who decide to punish the weakest of the weak, just because it's a convenient way to annoy me, are indeed worse people.

- The left should get back to talking about economics, yes. But I doubt that these voters would appreciate it, seeing as the last time the left made the case against neoliberalism, most of them probably complained about "the smelly hippies who should get a job instead of taking about some 99% or 1% BS"

- there are quite a few improvements for a factory worker who lost his job that the left has implemented, or would like to if not for a complete boycott of government by the right: universal healthcare, universal income, consumer financial protection (payday-loan and similar specifically targeting poor people). Strong anti-trust regulation. Redirecting money from the military to investments in infrastructure/education/health...

Politics is not just about being right. It's about convincing others you are. That's where the left has failed (if you agree with their positions).

Agreed, the left has real ideas for solutions to middle America. But these ideas are not what people want, so the burden is to prove that they work. Unfortunately for many of those ideas, such as universal income, it is still too early to say. Meanwhile other ideas, such as the current implementation of healthcare, is actively being rejected by voters after it was tried, i.e. it does not work (they're like a product customers do not buy, even if it's good for them).

You're also right that Trump's voters all not all poor and living in trailer parks (they also weren't all old people). I haven't seen data on socioeconomic background, but I wouldn't be surprised if they were essentially middle of the road. But for good reason, they are afraid for their economic futures. It's a justified fear, and it's good cause for feeling desperate.

Unfortunately for many of those ideas, such as universal income, it is still too early to say.

As others have noted in this subthread, that's not a desired solution for many of these people. They want to earn their money, not "get a handout".

Look at McGovern's 1972 campaign "Demogrant" proposal, $1,000 to every citizen, that's 5,750 in official 2016 BLS CPI dollars.

As I saw them at the time, the most effective political ads against him showed toy soldiers, ships and I assume planes, on a board (maybe of world?), and then a hand wiped off a large fraction of them, putting into visual language his (anti-)defense proposals, which I'll note weren't copied by Jimmy Carter, even before the Soviet invasion of Afghanistan.

But the second most was on the Demogrant, although all I can remember about it is some iron worker, maybe eating his lunch, high up on the side of a building frame, and I think it was in black and white.

I don't think enough Americans have changed, especially the Rust Belt ones that carried him over the top, that universal income is going to fly with them. That you think it has a chance suggests one of us is not in touch with these people. Then again, I find the idea personally abhorrent for what I believe to be the same reasons, but on the third hand I'm pretty old, at least by HN standards.

Oh, I definitely agree. I am not a UBI supporter by any means. Still, I think if the left can prove that it works on a smaller scale, and change people's mind, I'd be willing to reconsider (which is my default position on almost anything -- prove it and I'll believe it). But it's a long shot.

I don't think this has any relation to Big Data. It's simply Trump supporters were clever enough to give out false signals due to being ostracized by media and worried about consequences of going public (fired from a job, removed from social circles etc.) So the massive pressure on conformity created a late-Soviet-Union-style dissonance, outwardly many supporting "approved" candidates, inwardly supporting Trump (maybe due to the this pressure from all sides as a simple vote against). Many would've probably voted for "nobody" if that were a choice alongside Hillary and Donald.

A) This is NOT Big Data

B) This election was 101 in ML. The biases were obvious and easy to undo.

C) I called it and won a lot of money https://news.ycombinator.com/item?id=12863445

D) Best not to take advice from people who are constantly wrong

E) Big Data and ML are still awsome :)

See below from my linked post a week before the election:

The problem with Bayesian is that it's often over confident in the model. Error bars don't take this into account and people over rely on error bars - especially with complex domains. Then there are the intentional and unintentional sampling biases in the polls that have to be taken into account. Then there are known human effects that add additional biases. E.g. the shy voter effect (e.g. Torres and Brexit), silent majorities (Nixon), momentum, dam breaking (Reagan), enthusiasm gap (Obama), the giant fk you to the establishment vote, pending indictments etc. In my view with the election being so emotional this year these effects are big enough to swing it to Trump making a poll based model wildly inaccurate.

I've taken an actual position in the election. $4k on Trump to win at 15% odds made just after the tapes. At the time I felt Trump had at least 50% chance so the bet was positive expected return for me. I figured the public would get over it. My Bayesian friends had his odds at 2%. Today I consider Trump to be at least 70% with the whole FBI inditement saga picking up steam.

In addition, the betting market behavior mirrors Brexit. A few very big bets on the status quo and many small bets against. It appears as if once again deep pocket punters are intentionality trying to manipulate the odds on the illiquid market in order to send a message that effects the vastly larger financial markets. So I think the odds are that there is some free money there.

NOTE: The link only appears to work for my old account?? Was I shadow banned? Perhaps banning decenting opinions are a big reason why this election was such a supprise to everyone.

https://news.ycombinator.com/threads?id=ttyltrful

Not banned / save the conspiracies for another day / "dissenting" / "surprise"

Then where is it? Why are my posts missing?

It does look like ttyltrful was banned.

https://news.ycombinator.com/item?id=12684780

If you enable "showdead" in your account settings for huffpopo, you should be able to see them.

As for banning dissenting opinions, watching the behavior of sctb and dang, it's pretty clear to me that they ban based on uncivil behavior, not content.

Users on the other hand do sometimes flag or down vote to express disagreement.

Thanks. Makes sense, I was AB testing the community.

I'll be quitting HN now. I won't bore anyone with details. All the best.

Yes, your previous https://news.ycombinator.com/threads?id=ttyltrful appears to be banned. Likely, it was after you ignored Scott's warning here: https://news.ycombinator.com/item?id=12684066. Statistically, it's probably a good decision to ban new accounts that start off with two snide comments.

The particular comment you linked (https://news.ycombinator.com/item?id=12863445) seemed reasonable, so I 'vouched' for it. It doesn't look like that was enough to bring it back from the dead, though. You (or anyone else) can see it if you turn on the "Show Dead" option in your profile.

Thank you

EDIT: I should point out that the second snide comment was sarcasm on a topic using ML to detect sarcasm and not intended to by snide. Just a little joke for those of us who have worked on this problem.

Congratulations on the money, but it doesn't really proof anything, especially not (B). There were hundreds of people working on this, considering all sorts of possible sources of errors. Any bias that is "obvious and easy to undo" was priced in.

And the polls were actually not as bad – the difference was below the 3% from 2012. It just happened to swing the result.

By "obvious" I meant to ML practitioners (esp frequentists which is more typical for big data). Not everyone. So not priced in. It is not an efficient market. There are huge information asymmetries. I will give you that calling it ML 101 was over the top.

If you dig into the polling breakdown and compare the individual polls (states as well as national) with the actuals it is clear that the polls were bad. Just because the aggregate error was less off than it otherwise could have been is more luck than anything. It could be argued that you can assume a gaussian error distribution so a more precise aggregate is to be expected but I would argue that that is not a safe assumption.

TLDR; Just because it was close doesn't make it a good model.

As an aside. All of my Bayesian friends lost money and all of my Frequents friends made money :)

The Fourth major failing is in assuming that the majority of the country faces the same concerns as the rich people piled into the coastal cities.

There's nothing inherently wrong with calling upon like-minded people to change government policies as the author asks (it's called lobbying), but the assumption that the "other-half" are just mistaken bigots and racists is exactly how this mess started.

Exactly. Frank Bruni over at the NYT wrote this yesterday:

"Liberals miss this by being illiberal. They shame not just the racists and sexists who deserve it but all who disagree. A 64-year-old Southern woman not onboard with marriage equality finds herself characterized as a hateful boob. Never mind that Barack Obama and Hillary Clinton weren’t themselves onboard just five short years ago.

"Political correctness has morphed into a moral purity that may feel exhilarating but isn’t remotely tactical. It’s a handmaiden to smugness and sanctimony, undermining its own goals."[+]

And note too that some of us in tech who went to big-name private universities, maxed out our SATs, are good at math and love the physical sciences....... some of us "held our noses" and voted for Trump as well, finding, in comparison, the politics and platform of Hillary Clinton to be alienating and slightly worse, on the whole.

[+] http://www.nytimes.com/2016/11/13/opinion/the-democrats-scre...

It does seem like a smart guy with a twitter account, an airplane and a lot of energy was able to beat large embedded organizations and large infrastructures. Maybe it's like a nimble startup overtaking a large status quo industry. Some people say that his party actually had a more sophisticated big data system, though. It's just that they used it, instead of talking about it.

I urge everyone to read Social Physics by Alex Pentland, it addresses some of the topics in the article and provides a more holistic mental framework for big data. It's a good read. https://www.amazon.com/Social-Physics-Networks-Make-Smarter/...

Clickbait.

The TLDR:

A) Polls were biased and we don't have many elections on which to build good models.

B) Social media created echo chambers.

C) Other people's politics are sci-fi and more people should agree with me.

I stopped reading after the first few sentences that were politically driven. Click bait?

Wow, talk about a click-bait headline... absolutely nothing to do with big data.

Reply on news.ycombinator.com