mendicantB

It's dangerous to assume an even spread. Numbers are reported like this because the distribution of site visits is usually confidential information. From experience, I'd guess it's something like 1 million sites holding ~70% of the visitors, with the rest spread along the tail.

I didn't wish to bring in Bayesian approaches here so as to keep it simple. You are correct in your assertion about likeliness given a priori intuition, and I agree with the linking.

But, there is a stark contrast between leveraging prior information to strengthen your estimates and stating that prior information serves as evidence or correlates with outcomes, which was my original assertion.

Either way, I'm a big fan of leveraging common sense (viz Bayesian approaches) so I can't disagree.

Statistics is the means of testing whether or not something is evidence.

I understand what you meant, it's an intuitive conclusion, one that doesn't seem like magic. But, something making sense usually does not correlate with or serve as evidence of it being true.

You're not the only one. I think that strategy is overzealous and has potential to have terribly unintended consequences. Adria Richards comes to mind.

These algorithms are implemented in just about every major language, as they are standard tools. I'd say your best bet for leveraging them is python scikit learn (super good documentation)

I'm not sure how anyone can take this douche seriously after the leaks.

You disproved your own point.

In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation).

In statistics and probability theory, the standard deviation (SD) (represented by the Greek letter sigma, σ) shows how much variation or dispersion from the average exists.

I think you're missing the difference.

It's correct that a normal distribution is assumed in physics, but you are not correct that a standard deviation is defined in terms of a normal distribution.

He's correct. The poster he was responding to made an idiotic statement:

"In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)."

He was just stating that the standard deviation is a general concept of spread, that any distribution has, and not just the normal one.

I interviewed there (didn't get an offer, but agreed with the decision, felt too limiting technically).

In practice, it's a company wide pissing match that seeks to incubate and accelerate growth for hyper-achievers with hard heads that can openly give and receive criticism. This actually works extremely well for a certain type of personality (think Type A), especially if you are early on in your career. The growth potential was why I interviewed in the first place. I'd say the most accurate summary is:

http://www.businessinsider.com/inside-bridgewater-what-emplo...

Inherently, this type of culture results in one major drawback; lack of diversity. If you need a specific type of personality in order to join a company of others who have very similar personalities, what happens to innovation? Where do new ideas come from? How do you evolve? How do you understand different markets and audiences? etc etc. The best teams I've seen have had very diverse sets of backgrounds, level of experience, and personality.

While BW has worked out much of this (being in finance helps), that was my burning thought throughout the process. My conclusion was that such a culture would be an excellent place to be for some time, but staying too long would only serve to pigeonhole you.

So spot on. Considering recent privacy issues, Github needs to be careful to respond on this. It has potential to explode into an entirely separate shitstorm.

Don't forget to add shake you down for cash vs hit a button on your phone.

Bayesian approaches are probably out of grasp for most small companies. They have a long way to go before being as approachable and easy as frequentist approaches. Schools and the statistics field as a whole need drastic reformation in introductory course offerings that are taken by everyone.

Until then, it's A/B, p value <.05, ignore bias and sample size for companies who aren't large enough to have a statistician or data scientist.

It's not about developing an alternative, which will likely flounder anyways, considering that WhatsApp has the momentum and critical mass on users already. It's completely a defensive mechanism. WhatsApp dominates and will continue to dominate markets FB chat could never touch, and may invade current FB chat territory.

I felt the same about the YC ecosystem, which is why I submitted this.

I'm glad you're being open minded about responses.

While I completely agree with you on the massive backlog of problems that could be solved with very simple data analysis (this is the reason I am in the field), your thoughts about data modeling techniques are very much off the mark and sound like something read from a dated statistics textbook.

First, on logistic regression:

-I can tell you with certainty that logistic regression is not vastly underused, and is consistently among the most popular machine learning algorithms [1]. From personal experience I think it's likely the most powerful model there is.

-People are excited about Bayesian models and NN because they work. Actually, most logistic regression in practice is Bayesian by nature (regularization much?). More so, logistic regression is actually a piece of what powers NNs, and it's incorrect to directly compare the two. To put it more blunt, these are drastic improvements to logistic regression that allow the technique to solve greater problems.

-You are trivializing the amount of training needed to properly apply a classifier by calling it simple.

Second, there are plenty of basic questions in the data field to answer [2]:

-Doing anything with large amounts of data is still complicated and hard

-Unifying multiple data sets

-General model comparison

-Most modeling assumptions and as a result conclusions reached from them are wrong

-Communicating results to people. This is the most underestimated. One of the reasons we have such a backlog of problems that haven't been touched by simple data analysis techniques is the difficulty involved in explaining results. P-values are fucking meaningless and massively misinterpreted and misused. I won't start on frequentist statistics.

1) http://www.quora.com/What-are-the-top-10-data-mining-or-mach...

2)http://normaldeviate.wordpress.com/2012/06/21/90/

You are correct that validation sets are only one technique. But, the concept of validation, and the reasoning/justification to do it is an absolutely central idea. What is the point of a model that doesn't aim to generalize?

Honestly, calling validation a trick isn't helping.

Understanding the motivation behind validation is an absolutely fundamental concept, and lack of coherence on the topic shows an inherent lack of understanding of the goal of building the model in the first place; GENERALIZATION.

This is synonymous with one checking in code that has no issues locally, without testing in the stack or a production environment.

I work and hire in this space and it's actually a bit shocking how widespread this lack of understanding is. Asking a candidate how to evaluate a model, even at a basic level, is this field's version of FizzBuzz. Just like Fizzbuzz, a lot of candidates I've encountered who are "trained" in machine learning or statistics fail miserably, and my peers seem to have similar experiences.

These issues are expected, given how popular data science is these days. We all win when more people are getting their hands dirty with data, but it's extraordinarily easy to misuse the techniques and reach misleading conclusions. This can potentially lead to people pointing fingers at the field and it's decline. The only thing we can do is correct the wrongs and do our best to limit incompetence that only serves to tarnish the field.

My point exactly. There are numerous situations that could doom their product.

Looks like he took your advice. Regardless, I still got the same impression you did.

Sure, in that possibility, that is very true in that nobody could build a full fledged knockoff product. But, what concepts or features that could result in cheap knockoffs? Designed attacks? Password leaks and user privacy breaches? Customer information that can be sold to competitors? All of the bad PR and loss of business as a result?

To be frank, this isn't some minor display bug, he had access your source.

In other words, this could have ended your company.

He could have sold or leaked it. If naivety is stopping you from grasping the possible consequences, then go ahead and read about Adobe's recent mishap.

You should be on your knees worshipping him.

Yes it is actually. He pointed out the obvious between the lines message.

I think your point is very fair in that the author is exaggerating. But,

"highly readable"

Are we talking about the same language? The only reason R ever saw use is it's adoption by the statistical community, but all other things considered, R is a shitty language.

-Awfully difficult syntax

-Extremely high learning curve

-Nightmare to debug (error and warning messages are the most cryptic I've ever worked with)

-Some of the worst documentation I've ever encountered

-Many transformations are not visible to the user (how special characters are handled for example)...

-The largest set of statistical libraries...but have you ever tried using any? Good luck! Next to nonexistent documentation for most, and even with a background in mathematical statistics, I still have a hard time following examples

-More often than not, lack of backwards compatibility

I'm a statistician by training and data scientists by day job, and have used R for years. I really admire what some have tried to do with the language (Hadley). But, it's a result of the statistical community being light years behind on computational training and is by no means a good language; it just happened to come along at the right time and had some really genius initial developers.

Frankly, the "language of choice for data science" simply doesn't exist yet. You just use what works best for the problem at the time, and more often than not that involves switching between many languages.