Hacker News Reader

Comment by icelancer | Hacker News Reader

Both lead to common biases - the former in fraudulent experimentation, the latter (your suggestion) in fraudulent data snooping and anecdotal case-finding. Both have their merits done correctly, however, but the latter is only possible because of the former - the data has to come from somewhere, and experimentation was how it was historically done... since there wasn't a global database upon instantiation of the planet.

View on HN

I argue that there is no such thing as formulating a hypothesis with no prior knowledge. We all have prior observations - hypotheses don't appear out of nowhere.

In this respect I think GP's comment is admirable - it means teaching kids to use their experiences to formulate hypotheses, rather than forcing them to guess what you want them to formulate.

(As an aside, I set out to comment here because I tire of the point that XYZ can be done incorrectly... Everything can be - it's not very interesting - I wish we could discuss less about that, more about new things to do correctly.)

Formulating hypothesis first, even with little knowledge, is fine as long as the point is to invalidate them (basically you assume they are wrong and you’re fine spending time to prove it).

It can help focus the data gathering phase, and you get two results: you can ignore your first hypothesis from now on, and you have further data to look at to formulate a decent hypothesis to expand on.

I agree in practice, but I think my point is to resist the dichotomy you are implying - that there is a "first hypothesis with little data"... vs ... the other ones..?

Isn't all our knowledge little knowledge - even if it seems like a lot? Wouldn't your advice apply equally as well to your most knowledgable topic?

The hard part isn't having knowledge, it's finding the hypothesis to show how your knowledge is little knowledge.

(Aside: I would love to hear the experiences you are forming your comment around! I'm sure there's a lot of interesting stuff there.)

I think we fundamentally agree, and probably have the same approach, with just a different “marketing”.

For the background of these, on a professional level I try to sell the experiments with some angle, whatever it is, even if it is intuitively wrong.

An exemple of that was a discussion around wording for a relatively delicate part of our site. We didn’t know how to approach it, and it hasn’t been touch for a very long time. So, the random hypothesis taken was that nobody reads the text anyway. We set hooks and tracking in and out of the sections to have a decent view of what’s happening, and we got enough data to come with new insights and propositions; Next phase was AB testing two changes, with expectations to have positive effect, and the tracking we set in place was to validate it went as planned. This also didn’t go fully as expected, but not that far from the mark, and we iterated.

In hindsight we could surely have just done the data collection and make hypothesis from it, but I feel the data would have been more vague and less exploitable. I don‘t trust “let’s just collect data” approaches much.

IANAPOSE (I am not a philosophy of science expert) but I think a happy medium is to observe, collect data, and look for things that are interesting, and then perform properly formulate experiments to test those hypotheses.

I think that focusing on either side alone can lead to biases and problematic science, and focusing on both is actually the most "fun"

Both center around observation for all definitions of the word. Before we had microscopes or MRI machines, we still have those but we now have powerful statistical machines called ML models. In all cases though, we need perceive the world.

You can't see what do you don't look for.

Reply on news.ycombinator.com