Statistical shortcomings in standard math libraries (and how to fix them) | Hacker News Reader

Statistical shortcomings in standard math libraries (and how to fix them)

evanmiller.org

85 pointsjwmerrill19 comments

Unfortunately, it looks like the Cephes library of mathematical functions linked from this article is released with a poorly specified license, and contains code that is copyrighted by multiple organizations: http://www.netlib.org/cephes/readme

A bit of Debian discussion and interaction with the author on this: https://lists.debian.org/debian-legal/2004/12/msg00295.html

The GNU Scientific Library (GSL) covers these functions and is GPLed: http://www.gnu.org/software/gsl/

GSL's design doc (https://www.gnu.org/software/gsl/design/gsl-design.html) explicitly names many libraries, including CEPHES, that its creators consider useful but with drawbacks, such as non-free licensing, that motivated GSL's creation.

I'm not especially familiar with GSL except for a few functions I've used on occasion, and not with CEPHES at all, so I can't speak to other pros or cons.

GSL is pretty fantastic though it's not insanely high performance and doesn't accommodate extended precision types.

[Edit: I overlooked the fact that, being free under GPL, the GSL implementations do not suit the author's intent. (See /jwmerrill's comment below). Leaving the comment anyway just for reference, but my original point is moot.]

This. The author should have at least mentioned the ubiquituous and reasonably well-maintained GNU Scientific Library (GSL) [1], which already implements all the functions listed in the article [2,3,4,5], among many, many other things [6].

[1] http://www.gnu.org/software/gsl/

[2] http://www.gnu.org/software/gsl/manual/html_node/Incomplete-...

[3] http://www.gnu.org/software/gsl/manual/html_node/Incomplete-...

[4] http://www.gnu.org/software/gsl/manual/html_node/The-Gaussia...

[5] http://www.gnu.org/software/gsl/manual/html_node/Bessel-Func...

[6] http://www.gnu.org/software/gsl/manual/html_node/

The gsl is a great resource, but I don't think it suits the author's purpose. He is talking about adding new functions to the C standard library, for anyone to use in commercial or non-commercial software. It isn't practical to stick GPL code into C standard libraries that are meant to be used broadly in commercial as well as free software.

You are absolutely right. Now I see why the author didn't mention it. That was a knee-jerk reaction on my part.

How is the cephes license non-free?

Presumably if the "C" committee decided to do this, implementers would be free to implement this by various means, whether it be a clean-room version, or by adopting existing code. In the latter case, they could contact the author(s) of Cephes and try to arrange more suitable licensing if needed.

Admittedly it adds a little extra friction to the process, but this doesn't seem like an insurmountable problem off hand.

The meat isn't in the footnotes, so if a mod could update the link it would be nice.

Sorry about that.

We are still using distributions in FORTRAN written decades ago to generate random numbers in a variety of distributions. Having support for the inverse standard normal CDF would be incredibly helpful, especially when I had to go back in and add support for correlated uniformly distributed variates.

I see that the author of TFA wants this stuff put in the various standard libraries, and I agree that that would be good. But in the meantime, at least there are a variety of libraries one can use for accessing statistical functions. In Java-land there is Commons Math[1] which provides a number of statistical functions (in addition to other numerical/maths functions).

Things like Quantlib and jQuantlib also provide a lot of mathematical/numerical algorithms. Yeah, it's a drag having to pull in another library, but at least these things exist. :-)

[1]: http://commons.apache.org/proper/commons-math/index.html

I wouldn't have guessed these functions were so popular. I think those people who need them, need them often, but they are very few, so standards committees just ignore their needs.

If I have time this summer, I'll implement the Gamma and Beta functions, and their inverses, with a permissive license.

The problem with this is that most programmers don't need statistics on a regular enough basis. That's how things get into standard libraries: people need these things so regularly that there's no point in reimplementing them forever.

You really think we need stats functions in libc more than we need, say, properly written accelerated vector and matrix functions? Which do you think comes up more regularly in practice?

You're better off reimplementing these handful of functions in a BSD-licensed copy-lib and forgetting about it. None of these are particularly tricky to implement.

> The problem with this is that most programmers don't need statistics on a regular enough basis.

Empirically, many language environments (think, e.g. scripting languages) don't have really easy access to all of these functions, but do have easy access to most of the rest of libm. The author is saying that if these were part of libm, then language implementers would include them in their respective standard libraries, and then people that care about stats (and many of us should care about stats!) could write good stats software in general purpose languages that we might have chosen for some independent reason.

> You really think we need stats functions in libc more than we need, say, properly written accelerated vector and matrix functions?

Can't we have both?

The (dense) vector and matrix functions have lived in Lapack and BLAS for 40 years, longer than there has been a libc. Every serious language has a standard interface for them, and the issues discussed in the article haven't caused problems for decades.

> The problem with this is that most programmers don't need statistics on a regular enough basis. That's how things get into standard libraries: people need these things so regularly that there's no point in reimplementing them forever.

True, although I suspect there's probably a certain amount of historical accident to the set of functions included in math.h. There's a strong argument that some of these common 'special' functions are at least as fundamental to a variety of mathematical fields as (say) arc hyperbolic cosine, although admittedly it'd probably be hard to get clear consensus on exact which.

I expect if you asked a variety of stats/ML geeks they might have other priorities to those given here -- e.g. I might request the digamma and trigamma functions (first and second derivatives of log gamma, useful for computing gradients in various Bayesian models) before asking for Bessel functions, say. It might also be sensible to prefer log-domain versions of some of the functions he suggests, since much practical computation with probabilities needs to happen in the log domain.

"None of these are particularly tricky to implement."

Almost every mathematical function is hard to implement, if you want an implementation that is correct and (about) as fast as possible. For example, see http://www.boost.org/doc/libs/1_55_0/libs/math/doc/html/math..., which explains how boost implements the Bessel functions.

(By the way, I think boost math special functions is that 'BSD'-licensed library you are asking for)

Reply on news.ycombinator.com