https://github.com/sradc/SmallPebble
The first line of the description is:
> SmallPebble is a minimal automatic differentiation and deep learning library written from scratch in Python, using NumPy/CuPy.
- it's very much taking the edsl approach
- it's only implementing forward-mode
As for the edsl, you can claim it is, but why? There is no parser, no syntax, nothing that would make any user think they are learning a new language. Would you call numpy an edsl ?
embedded dsls don't have parsers; that's what makes them embedded. numpy is solidly in the center of the embedded dsl concept
I see. It's actually worse than both forward and reverse mode.
Let's say you have the simple program x1+x2+x3+x4+x5. Internally the program calculates x6=x1+x2, x7 = x6+x3, x8=x7+x4 and result = x9 = x8+x5. In forward mode you get all the partial derivatives for all intermediate variables w.r.t the inputs x1 through x5, so you get things like dx8/dx2, but you never calculate (or store in memory) dx8/dx6. In reverse mode you get the derivatives of the result w.r.t. all intermediate variables, so things like dx9/dx8, dx9/dx7, etc, but again you never need to calculate dx8/dx6. In this library you calculate recursively all dxj/dxi and store them, as long as xj has a direct or indirect dependency on xi. This is a strict superset of the derivatives calculated in both forward and reverse mode.
Good catch.