syntacticsalt

As briefly as I can.

Typically, the reason we're running benchmarks is to detect changes of at least some minimum difference -- a minimum detectable effect size. Assuming such a difference exists, we'd typically want to detect it with a high enough probability (i.e., statistical power, which is 1 minus the probability of a false negative), or else we're wasting time and money on a flaky result. Given a target false positive rate, power, a minimum effect size, and a few other assumptions, we can estimate how big of a sample size we would need to drive down our false negative rate enough to make the benchmark a reliable detector of performance changes, and thus the resources we'd need to spend on running the benchmark for it to be useful.

These other assumptions typically rely on statements about what the statistical test is trying to detect. Phrased in the context of benchstat, a Mann-Whitney U test is testing whether the probability of a sample of A being less than or equal to a sample of B is (without loss of generality) more than 1/2. The benchstat docs state "The p-value measures how likely it is that any differences were due to random chance (i.e., noise). In this case, it's extremely unlikely the difference between the medians was due to chance." If we make an additional assumption that the distribution of B has the same shape as the distribution of A, except that we've shifted its location -- a location-shift alternate hypothesis -- then the Mann-Whitney U test is testing a difference in medians (sort of; I'm omitting details for brevity).

benchstat's confidence intervals on medians are unrelated to the U test, so we can't use them to estimate the probability of a false negative under assumptions -- assumptions that benchstat isn't entirely explicit about -- and it's difficult to provide a user with some feedback suggesting they may want to run more (or fewer!) samples. There's no reason to believe that a location-shift hypothesis for benchmarking is sensible, if that is indeed an assumption the package's authors make deliberately, and it's not entirely clear to me that the assumption is deliberate, either. A location-shift assumption could be reasonable in some contexts, but we don't know, and benchstat doesn't provide any diagnostics to tell us one way or the other. benchstat also doesn't really clarify why (or when!) n = 10 or n = 20 is a good idea. Unless the minimum detectable effect size we're looking for is large, I'd expect benchstat's approach to be underpowered, so it might fail to detect smaller performance differences. And if n = 10 or n = 20 is really a hard limit or an overriding user preference, there are methods that trade simplicity for a modest amount of compute, like permutation tests, that are simpler to explain and easier to use for simulations to determine statistical power without making parametric assumptions; these approaches would generalize to bootstrap or Monte Carlo methods at larger sample sizes, and permutation tests tend to have higher power than Mann-Whitney unless populations are skewed.

Medians are asymptotically normally distributed, so it's possible to use t-tests or bootstrap methods. There are caveats to such approaches, of course, but it's easier to do power calculations in that context to estimate prospective sample size, even if only to do back-of-the-envelope estimates.

The nonparametric approach benchstat uses to compute confidence intervals on medians is coherent, but at such sample sizes, it also tends to be coarse. If we're really concerned about the difference in medians as an engineering figure of merit, it would probably be a good idea to compute a confidence interval about that difference, too, but benchstat doesn't do that either, and if it's making a big deal about medians, I would argue it probably should. If relative difference in medians is the preferred figure of merit, a delta method correction for a confidence interval would be a good idea for that quantity.

As far as I could tell, benchstat seems to implicitly make a location-shift assumption in its interpretation of the Mann-Whitney U test, and I don't see why that assumption would hold.

Yes, I should probably clarify and amend my argument.

My impression of the article is that it suggests if ergonomic APIs were more available than they already are, then the frequency with which we see bugs in implementing random variable sampling would decrease. I question that premise because the ergonomic APIs already widely exist -- as the article itself points out, in many language standard libraries, and also in popular third-party libraries. Such a suggestion seems like it relies too much on a single mitigation.

To borrow your examples:

- we don't rely on safety belts alone to reduce injuries: we augment that control with more engineering controls, regulations, and education. - we don't rely on gun safeties alone to reduce injuries: we augment that control with more engineering controls, regulations, and education.

For obvious reasons, I think regulation or mandatory education would be impractical for this situation.

Because other engineering domains, including the ones you mentioned, rely on complementary controls to achieve further safety, I speculated that fuzz testing and deterministic simulation testing could be effective potential complementary controls. Property testing could be another.

I make this argument because I work as a computational statistician with software engineers in a large software company and I see variations on the theme of these bugs on a weekly basis. The usual justification I get is "I know that (insert library implementation) exists but I thought what I wrote was a simpler version of the same thing, and then I have fewer dependencies", as if equivalence of function were self-evident. It's not, and sometimes I can persuade the engineer to use the library from first principles, but generating witnesses violating the properties of the object they're computing tends to be more compelling, and generalizable strategies for testing tend to get more buy-in because the value-add is not limited to statistical applications.

Curious why you used Box-Muller for normal PRNGs instead of Ziggurat.

A Uniform(0, 1) PRNG is the right primitive on which to base a PRNG library for arbitrary real-valued random variables because any real-valued random variable can be represented as an inverse quantile transform of a Uniform(0, 1) random variable. While I agree that only providing this primitive risks footguns as described in the article, I'm skeptical that providing a better UI alone would meaningfully reduce the risk of such footguns because availability and convenience is no guarantee of use if a prevailing attitude of users is that they know better and don't need it. Bisection search is simpler than rejection sampling or inverse transform sampling, yet it's common to see buggy, hand-rolled implementations of bisection search despite wide availability of library implementations with better UI ergonomics than PRNGs. I think wider use of fuzz testing or deterministic simulation testing really is necessary to disabuse people of that notion, along with more articles like the above explaining why hand-rolling an adapter to a uniform PRNG is a false economy compared to proven implementations with vetted statistical properties.

I'm not sure I follow why your argument changes the relevance of convexity? If the problems were convex, I don't think we'd be having this discussion -- first-order methods would tend to win.

I'm skeptical as to whether zeroth-order methods really lend themselves to a Bitter Lesson argument. First-order methods don't explore the loss landscape optimally, but the loss function tends to be nonconvex, and zeroth-order methods don't address that issue head on. Dust smooths, and so do applicable first-order methods. Remove the nonconvexity issue, and I suspect Dust's purported advantages evaporate (based on published theoretical work), so it's pretty odd to me that the paper never discusses convexity.

I could buy that this method scales better than previous zeroth-order methods, and that's interesting, but it doesn't seem like enough of a moat to keep improved first-order methods from drinking its milkshake, except in cases where a zeroth-order method is already a primary option: the network needs to call a simulator that doesn't expose gradient-like information. (In cases where gradients don't exist, I'd still argue for other options, e.g., Clarke-generalized gradients where applicable, so long as those can be computed with the available information. I know this technology has been published for automatic differentiation, so I would imagine it could be incorporated into backprop and used with a suitable optimization algorithm.)

Even for non-smooth or discontinuous objectives, I'd still reach for methods that use gradient-like information over zeroth-order methods. For non-smooth objectives, Clarke-generalized subdifferentials have been pretty effective outside of ML, and have been used in automatic differentiation contexts at least 10-15 years ago. A carelessly quick literature search suggested conservative gradients, too.

For discontinuous objectives, I know there's been work on using envelope approximations, but the little I'm aware of in that work was in low-dimensional settings where the structure of the discontinuity was known explicitly. On the other extreme, lack of continuity comes up all the time in infinite-dimensional, PDE-constrained optimization, and some methods rely on tangent cones or various generalized notions of subdifferentiability (e.g., Mordukhovich, Bouligand) to demonstrate convergence. Admittedly, that work was somewhat outside my area of expertise, so I may be getting the details there slightly wrong, but the broad point stands that even in those settings, some directional information can be obtained and used profitably without resorting to zeroth-order methods.

Pixi uses uv as a backend, and I've enjoyed the UI because it's easy to add task aliases for things like nicely-formatted lists of outdated packages. (I have no affiliation with the project.)

Pixi-diff-to-markdown in particular has made scanning automated CI package updates easier. So for something like viewing outdated packages that would be updated, I'd do something like create a task alias for a project:

pixi task add outdated "pixi update --dry-run --json | pixi exec pixi-diff-to-markdown"

And then run the task in the project via:

pixi run outdated

The output is a readable Markdown table of packages that would be updated, old version, and the new version that would be installed using the pixi update command. Your mileage and tastes, of course, may vary.

I don't disagree about the complicity. However, biology and statistics, even at intro level, have had significant updates in material covered over the last 10-20 years.

More subtly, terminology changes. My copy of Rudin's Principles of Mathematical Analysis is just as correct now as it was when it was published in 1976, but I remember one of my professors describing the terminology as somewhat dated, as of the late 2000s.

My opinion of Gwern's piece is that some of the arguments he makes don't require correlations. For example, A/B tests of differences in means using a zero difference null hypothesis will reject the null, given enough data.

In that A/B testing scenario, I think if someone wants to test whether the difference is zero, that's fine, but if the effect size is small, they shouldn't claim that there's any meaningful difference. I believe the pharma literature calls this scenario equivalence testing.

Assuming a positive difference in means is desirable, I think testing for a null hypothesis of a change of at least some positive value (e.g., +5% of control) is a better idea. I believe the pharma literature calls this scenario superiority testing.

I believe superiority testing is preferable to equivalence testing, and in professional settings, I have made this case to managers. I have not succeeded in persuading them, and thus do the equivalence testing they request.

I don't think the idea of a zero null hypothesis is necessarily mathematically unsound. In cases like the difference in means, a zero null hypothesis is well-posed. However, I agree with you that there are better practices, like a null hypothesis incorporating a nonzero effect.

I don't entirely agree with the arguments Gwern puts forth in the Implications section because some of them seem at odds with one another. Betting on sparsity would imply neglecting some of the correlations he's arguing are so essential to capture. The bit about algorithmic bias strikes me as a bizarre proposition to include with little supporting evidence, especially when there are empirical examples of algorithmic bias.

What I find lacking about Gwern's piece is that it's a bit like lighting a match to widespread statistical practice, and then walking away. Yes, I think null hypothesis statistical testing is widely overused, and that statistical significance alone is not a good determinant of what constitutes a "discovery". I agree that modeling is hard, and that "everything is correlated" is, to an extent, true because the correlations are not literally or exactly zero. But if you're going to take the strong stance that null hypothesis statistical testing is meaningless, I believe you need to provide some kind of concrete alternative. I don't think Gwern's piece explicitly advocates an alternative, and it only hints the alternative might be causal inference. Asking people who may not have much statistics training to leap from frequentist concepts taught in high school to causal inference would be a big ask. If Gwern isn't asking that, then I'd want to know what a suggested alternative would be. Notably, Gwern does not mention testing for nonzero positive effects (e.g., in the vein of the "c > 0.3" case above). If there isn't an alternative, I'm not sure what the argument is. Don't use statistics, perhaps? It's tough to say.

Are you referring to the first figure, from Smith, et al, 2007? If so, I couldn't evaluate whether gwern's claim makes sense without reading that paper to get an idea of, e.g., sample size and how they control for false positives. I don't think it's self-evident from that figure alone.

One rule of thumb for interpreting (presumably Pearson) correlation coefficients is given in [0] and states that correlations with magnitude 0.3 or less are negligible, in which case most of the bins in that histogram correspond to cases that aren't considered meaningful.

[0]: https://pmc.ncbi.nlm.nih.gov/articles/PMC3576830/table/T1/

As you yourself point out, a consistent estimator of a parameter converges to that parameter's value in the infinite sample limit. That limit is zero or it's not.

A frequentist interpretation of inference assumes parameters have fixed, but unknown values. In this paradigm, it is sensible to speak of the statement "this parameter's value is zero" as either true or false.

I do not think it is accurate to portray the author as someone who does not understand asymptotic statistics.

Reporting effect size mitigates this problem. If observed effect size is too small, its statistical significance isn't viewed as meaningful.

I don't disagree with the title, but I'm left wondering what they want us to do about it beyond hinting at causal inference. I'd also be curious what the author thinks of minimum effect sizes (re: Implication 1) and noninferiority testing (re: Implication 2).

Permutation tests don't account for family-wise error rate effects, so I'm curious why you would say that "it doesn't overcorrect like traditional methods".

I'm also curious why you say those "cover every case", because permutation tests tend to be underpowered, and also tend to be cumbersome when it comes to constructing confidence intervals of statistics, compared to something like the bootstrap.

Don't get me wrong -- I like permutation tests, especially for their versatility, but as one tool out of a bunch of methods.

> Most companies don't cost peoples' lives when you get it wrong.

True, but it usually costs money to fix it. I think the themes of "this only matters if lives are on the line" or "it's too rigorous" are straw-men.

We have limited resources -- time, money, people. We'd like to avoid deploying those resources badly. Statistical inference can be one way to give us more information so we avoid using our resources badly, but as you note, statistical inference also has costs: we have to spend resources to get the data we need to do the inference, plus other costs. We can estimate the costs of getting sufficient data using sample size estimation methods. For go/no-go decision-making, if the cost of getting the decision wrong isn't something like at least 10x the cost of doing the statistical inference, I don't think it's worth doing the inference. It may be worth doing the inference for _other_ reasons, but those reasons are out of scope.

As an example, a common use of statistical inference in medical research is to compare the efficacy of a treatment with a placebo. Some of the motivation is to decide whether to invest more resources in developing the treatment, not because people will die if they get a false positive stating that the treatment is effective when it isn't.

> A lot of companies are, arguably, _too rigorous_ when it comes to testing.

My experience in industry has been the opposite. Companies like the idea of data-driven decision-making, but then they discover pain points. They should have some idea of how much of a change they're looking to detect (i.e., an effect size). They should estimate how much data they're likely to need to run their tests (i.e., sample size estimation). They have to consider other issues like model misfit, calibration, multiple-testing corrections, and so on. Then they also have to rig up the infra to be able to _do_ the testing, collect the data, analyze the results, and communicate the results to their internal stakeholders. These pain points are why companies like Eppo and StatSig exist -- A/B testing ends up being more high-touch than developers expect.

Messing up any one of these issues can yield "flaky tests," which developers hate. Failing to gather a sufficiently large sample size for a given effect size is a pretty common failure mode.

> But to "maintain rigor," we waited 6 weeks before turning it... and the final numbers were virtually the same as the 48 hour numbers.

It's difficult to tell precisely what you mean by "maintain rigor" here. The only context I can gather is that whatever procedure you were using needed more data in order to satisfy the preconditions of the test needed for the nominal design criteria of the test -- usually, its nominal false positive rate. I don't think this is an issue of rigor -- it's an issue of statistical modeling and correctness.

Sometimes, it's possible to use different methods that may require less data at the cost of more (or different) modeling assumptions. Failing to satisfy the assumptions of a test can increase its false positive rate. Whether that matters is really up to you.

> I do like their proposal for "peeking" and subsequent testing.

What the post is suggesting is not a proposal, but a standard class of frequentist statistical inference methods called sequential testing. Daniël Lakens has a good online textbook (https://lakens.github.io/statistical_inferences/) that briefly discusses these methods in Chapter 10 and provides further references.

> We're shipping software. We can change things if we get them wrong.

That's usually true -- as long as you have the resources needed to make those changes, and are willing to spend them that way.

> IMO, the right framing here is: your startup deserves to be as rigorous as is necessary to achieve its goals.

While I don't disagree with the sentiment, I think you're conflating rigor with correctness here.

> If its goals are "stat sig on every test", then sure, treat it like someone might die if you're wrong.

I think that's a false equivalence. Even the American Statistical Association has issued a statement on p-values (see https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf) that includes "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold."

> But if your goals are "do no harm, see if we're heading in the right vector, and trust that you can pivot if it turns out you got a false positive," then you kind of explicitly don't need to treat it with the same rigor as a medical test.

If those are your goals, just ship it; I don't think it makes sense to justify the effort to test in this situation, especially if, as you argue, it's financially feasible to roll back the change or pivot if it doesn't work.