Hacker News Reader

Comment by carlthome | Hacker News Reader

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs.

Is there some go-to practice material I could look at?

Splines I haven't touched since numerical computing exercises in school.

View on HN

Simon Wood's Generalized Additive Model book.

Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll.

I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.

> Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it.

Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.

If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness.

You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.

Reply on news.ycombinator.com