Hacker News Reader

Comment by jjoonathan | Hacker News Reader

I don't see any mention of automatic differentiation which I thought was the dominant technique in this area. Is there an advantage to the method described in OP (other than mathematical cuteness)?

View on HN

No. Automatic differentiation is superior to the complex step method in every way.

The only reason to use the complex step method is if you're using legacy languages where it's difficult to implement dual numbers but you have good support for complex numbers. I don't think anybody should be using the complex step method in new applications.

Some reading ~ http://aero-comlab.stanford.edu/Papers/martins.aiaa.01-0921....

wtf

Automatic differentiation only works for the simplest functions for which you already know what the Taylor series looks like. For those cases, you might as well just hardcode derivative functions and the basic derivative rules (linearity and chain rule). It is not a general-purpose method.

For functions that you can't even express by a simple formula, you still have to rely on finite differencing.

Don't call "legacy" anything that you don't understand.

Can you comment on the space of functions which you know to be complex differentiable but for which you can't analytically evaluate/approximate the real derivative?

If so, I look forward to learning about it. If not, you were throwing stones in a glass house.

Sure, a simple example is the derivative of Riemann zeta for Re s < 1. Granted, usually you work with the logarithmic derivative which is a lot more tractable, but if for whatever reason you need the actual derivative, you're gonna have a lot of trouble with automatic differentiation, as the typical analytic continuation formulas involve some complicated improper integrals.

There are also functions that you only know by sampling (e.g. ocean temperatures) for which you assume smoothness. You need to pick an interpolation method, but sometimes you do not interpolate beyond the sampling points, because that's just making up numbers. When you're limited by your original sampling step size, you have little recurse but to compute derivatives by some finite differencing scheme.

The question was about automatic differentiation vs the method in the article (the complex step method) which is essentially a defective variant of automatic differentiation with a spurious numerical parameter.

Automatic differentiation works for any composition of those ‘simplest functions’ you mention, which is quite a lot of stuff, including whole programs.

Approximation methods have their place, sure. Sometimes they're good enough and sometimes it's all you can do. What does that have to do with anything?

Thanks for the link, btw. I'm in the process of digesting the paper.

I was objecting to the suggestion that the complex finite differencing scheme was worthless ("legacy") and automatic differentiation is all we should ever do from now on.

He did link to a book of papers largely about automatic differentiation, http://amzn.com/0898713854 (table of contents here http://bt.pa.msu.edu/cgi-bin/display.pl?name=sfbook)

I already know that a book on numerical analysis is likely to contain the answer I seek. The point of asking was to avoid a trip to the library since the working memory of someone who has been investigating differentiation techniques also likely contains the answer I seek.

I don’t quite understand what your question is, so I’m going to assume you were trying to ask, “When would you not want to use automatic differentiation?” I think the main time you can’t get much out of automatic differentiation is when your function is a black box or approximation, e.g. it comes from measuring physical data, or is the output of some kind of simulation, etc. I’m not an expert though.

You can get a lot of hits if you google related keywords (e.g. a query with both terms "numerical differentiation" "automatic differentiation" in quotation marks). But here’s a start, http://alexey.radul.name/ideas/2013/introduction-to-automati...

You can probably get useful advice if you ask on http://scicomp.stackexchange.com or similar.

Someone who knows to bring automatic differentiation into a discussion about complex-step differentiation with the implication that they have already compared them and found one wanting (while implying that they may have missed something and are receptive to contrary opinions) already knows how CSD and AD work. This person also likely know how to use google, the library / amazon, and stack exchange (at the very least) but upon running a few simple queries has determined that the approximate effort to find an answer exceeds their interest in the subject. Unless someone else can use their own greater familiarity with the subject to provide an opinion or a pointer to a specific easily accessible resource (example: lloda's pdf link, counterexample: your book link) the line of inquiry will fizzle out due to the tyranny of cost/benefit analysis and having better things to do.

I appreciate that you were trying to be helpful, which is why I'm trying to be gentle (and explanatory) as I tell you that you weren't.

Reply on news.ycombinator.com