Is it signups that matter? Or DAUs? Or MAUs? Or (and it actually is this one) revenue?
But how do you decide the relationship between each of these? Does an easier on-board mean that you get less "sticky" users? How will you tell?
Constructing experiments is hard, and interpreting data is even harder. PhDs, whose job it is to do these things, who typically have something on the order of a decade of education and even more years of practice doing these things, PhDs mess these things up. It happens all the time.
That's not even getting into the fact that in many times in data driven decision-making, an experiment is constructed to motivate a particular decision. If you have a preferred outcome, it's very easy to deliberately or unconsciously put your hand on the scale. Even without deliberate sabotage, there are many methods to apply to a particular dataset, and if you apply enough tools, sure enough one will support your assertion.
More often than not the result is that data driven decisionmaking is more like an elaborate ouija-board that says more about the people creating the experiments than what they purport to verify.
After we hit that size we would analyze the results. If they were ambiguous we just picked the one we subjectively felt was the better choice.
We also used data a bit less formally. For example, we had something similar to Google analytics' "funnel" feature of how people arrived at some of our pages, and we noticed a usage pattern that indicated we were missing a feature. When we added the feature directly it was one of our most used features.