Comment by erikpukinskis | Hacker News Reader

Comment by erikpukinskis | Hacker News Reader

Parent Full thread

Edit: I can see this is getting downvoted. If you have time, I would really appreciate a response to go with the downvote. This is my heartfelt belief, and if I'm wrong I would really like to be able to learn something.

> Make you wonder if our industry's mind share being wasted on web apps and consumer gadgets because of the profit incentive associated?

I find this sentiment strange. To me, your kind of excitement, which seems to be pure joy over incredible engineering feats, seems nihilistic. There are serious terrestrial problems happening right now. Massive environmental destruction, humanitarian disasters. Even in relatively stable places like the Bay Area I know many people who are teetering on personal disaster. These are extremely complex problems and engineers have extraordinarily powerful skills to help solve them.

I understand why people are captivated by things like comets and why they'd want to attempt these extraordinary feats. It's exciting and fun and there are few joys that can match doing something that's never been done before.

For that reason I don't begrudge anyone who chooses that kind of work. Earth's crises are incredibly depressing and there's largely no glory in helping a few people stay out of homelessness, or helping a kid get through college. It's risky, demoralizing work.

But I don't really understand why a mission like this would be considered a noble pursuit. Fun, sure. But not especially noble. Yes, basic space science has some value to society in the long term. But we have a MASSIVE backlog of basic science that is being unused. We have journals chock full of solutions to problems that are going unimplemented.

Even something as simple as logistic regression, which has been around for 70 years, is vastly underused. People get excited about bayesian models and neural networks and all this great research, but there's still a massive backlog of problems that could be solved with a simple regression or an ANOVA. There are no basic questions to solve, just the work of obtaining and cleaning data, doing the analysis, and packaging it in a tool that people can actually use.

You seem to wonder why more people don't take on extremely difficult challenges on the edge of possibility. But I wonder why more people don't use existing technical tools to solve the massive backlog of problems that are being ignored.

I'm glad you're being open minded about responses.

While I completely agree with you on the massive backlog of problems that could be solved with very simple data analysis (this is the reason I am in the field), your thoughts about data modeling techniques are very much off the mark and sound like something read from a dated statistics textbook.

First, on logistic regression:

-I can tell you with certainty that logistic regression is not vastly underused, and is consistently among the most popular machine learning algorithms [1]. From personal experience I think it's likely the most powerful model there is.

-People are excited about Bayesian models and NN because they work. Actually, most logistic regression in practice is Bayesian by nature (regularization much?). More so, logistic regression is actually a piece of what powers NNs, and it's incorrect to directly compare the two. To put it more blunt, these are drastic improvements to logistic regression that allow the technique to solve greater problems.

-You are trivializing the amount of training needed to properly apply a classifier by calling it simple.

Second, there are plenty of basic questions in the data field to answer [2]:

-Doing anything with large amounts of data is still complicated and hard

-Unifying multiple data sets

-General model comparison

-Most modeling assumptions and as a result conclusions reached from them are wrong

-Communicating results to people. This is the most underestimated. One of the reasons we have such a backlog of problems that haven't been touched by simple data analysis techniques is the difficulty involved in explaining results. P-values are fucking meaningless and massively misinterpreted and misused. I won't start on frequentist statistics.

1) http://www.quora.com/What-are-the-top-10-data-mining-or-mach...

2)http://normaldeviate.wordpress.com/2012/06/21/90/

When I say logistic regression is underused, I don't mean underused among data scientists. We have an incredible shortfall of "data scientists". What I mean is there is data sitting on computers somewhere, which is a logistic regression away from giving someone some information that they could use to make better decisions. But there's no one at Debbie's Diner who is capable of doing the regression, so it goes undone.

And the computer scientist who eats Debbie's waffles every Saturday spends his weekends reading about SpaceX instead of pulling her data into R and giving her some insights that would give her a much safer retirement.

As a contrived example to my thinking: If it had taken us another 100 years to figure out combustion engines would we all be riding in steam cars? Would the industry focusing on building better and better steam cars have been a waste? Would investing millions in developing an engine that was light enough to allow flight in heavier than air vehicles when we already have a dirigible/train/ship network that is perfectly adequate for passenger and good transport. Personally I see this kind of work as needing to find 'new' solutions to realise these kind of blue sky projects. Meanwhile applying ourselves to even quite meaningful works does not give us a new solution to apply to other problems it just fits an existing solution to an unsolved problem. It may benefit us more in the short term to apply the solutions we already have (some times in very novel ways) to solve existing problems but coming up with new ways to apply the same old solution to different areas becomes harder and harder as there are less problems it could apply to and the blue sky research may very much make those solutions, if not obsolete, an inefficient use of effort.

I feel like a lot of the stuff I do is that sort of basic work, and that's just industry (old plant stuff). But there's value to taking advantage of some of the new tools. Occasionally they're massive force multipliers or reframe a problem in a more solvable way. For example, this year I plan to play with Beysian analysis - I want to be able to trace the story of process shifts, not merely detect them. I could - and will - hit the problem with the standard fare of tools, but it's an overwhelming amount of ground to trudge through. Just trying to make the imperfect, inhomogenous data analyzable to automate it is hard, and yet still worth the effort even in the short run compared to manually.

Back to space. There are some long shot wins with research like this, things that are pure research prerequisites for big changes. As in, once we've looked at and followed a comet, we'll have the experience doing remote landings on small rocks in space. Why not do that to asteroids? Perhaps we can find an asteroid with an unusually high concentration of rare earth metals. That could bootstrap fuel cell research by dropping the price of rare, useful materials like platinum. Or gold. Such tech would help alleviate many problems at once. It might break economies, but it may also put us post-scarcity for manufacturing tech.

I don't mean to make it sound like I don't think it's not worth doing just for its own sake. I really do. But I also think that the potential wins we're setting up for the next generation or three down the road make starting now worth it. Being slow and thorough is a risk of lost opportunity, potentially huge loss. I also think there's plenty of brilliance available to work on the other problems, too, and I am often confused that we feel a scarcity in that respect (like how there are food shortages despite incredible productive ability - I just can't properly wrap my head around the subtle complexities that cause such strange things).

So I think you have a point, but I think it's a bit harsh to call it nihilistic. That's certainly there, but there's more good to be had, too. Can we do better? Sure, but I'm cynical enough that I'll accept the occasional wins for what they are, while secretly hoping it changes to world for the better.

In order for the human race to survive, we need a foothold in space. The Earth, or even our solar system, won't remain habitable forever.

There's no hurry though, waiting 100, or even a million years won't have any impact on that, right?

And waiting a couple of generations won't matter re:

>Massive environmental destruction, humanitarian disasters.

Right?

That isn't anywhere near enough to cause extinction, and the vast amounts of resources, engineers, scientists, etc required to do that would be much better spent solving those problems instead.

It absolutely will matter, because in the course of those generations, people will die or suffer or both. I'm talking about triage. We don't need to worry about the sun exploding just yet.

Regarding cosmological expansion, a million years has virtually no impact on us reaching ~4.8 billion galaxies at 0.99c. [1]

[1] https://www.youtube.com/watch?v=zQTfuI-9jIo#t=1670

You saw what that tiny 20-meter asteroid did to that town on Russia last year right? this one http://en.wikipedia.org/wiki/Chelyabinsk_meteor

How many time do you think you have?

In that case it would be more rational to spend money on asteroid detection and deflection. That would be orders of magnitude cheaper and easier than building a completely self sufficient space colony.

That's equivalent of patching a legacy code base and delaying the problem not exactly fixing the root cause.

Your self-sufficient space colony can still be destroyed by asteroids too, so that doesn't solve the problem either. Deflecting asteroids is entirely possible so it isn't a "patch" either. And building a self-sufficient colony would be so ridiculously complicated and expensive, it might not even be possible at all. That sounds more like a "patch".

The whole human race is currently running off one hard-drive that has been in EarthPC for 10000s of years.

Downvoted you for asking people to explain their downvotes, which I consider a dark pattern.

I have to say I agree (since you are now being downvoted). The downvote is a quick way of indicating sentiment, if people have something to say they probably will.

Talking about the general pattern, not this case in particular:

I disagree. A downvote is a signal, "fewer comments like this one please". Sometimes it's obvious what that means, but sometimes it's not. In that case it's perfectly reasonable to ask, what was it about my comment that you disliked? A feedback signal is only useful if understood, and sometimes to be understood it needs clarifying.

I agree that it's a dark pattern, but I wish that more people would reserve downvotes for comments not constructive to the unpacking of ideas and opinions, such as inflammatory, derailing, or factually misleading/incorrect information. I feel that I've seen too many unexplained downvotes placed seemingly because of disagreement with a commenter's articulated contrarian opinion, and that's bad for meaningful discourse.

Reply on news.ycombinator.com