Reverse-mode algorithmic differentiation using effect handlers in OCaml 5

It looks like it's doing only addition and multiplication. The rest is left as an exercise to the reader.

Here's a much better introduction, that links you to an industrial-level autodiff library [1].

[1] https://sidsite.com/posts/autodiff/

Add and Mult are added as extension of the type Effect.t and run installs the effect handler. To add more operations you would have to extend Effect.t again then setup an handler somewhere.

The current effects syntax is very rough. It’s considered an experimental feature by the compiler team. Currently effects handlers are exposed to users as a thin wrapper on top of their implementations in the runtime. A better syntax will come but the Ocaml dev team wants to take the time necessary to get it right.

Automatic differentiation gives you the ability to freely define the forward propagation of your neural network and you get backpropagation for free. The NN library "Flow" for Julia makes great use out of this. Having automatic differentiation makes it very simple to define novel layers, with very little work.

>is it ever required for applications to implement their own differentation for something specific?

If you want to do backpropagation you need to manually or automatically calculate derivatives woth regards to your parameters.