https://github.com/983/Elephant/tree/master
Seriously, that's very cool. Well done.
I've tried to add neural time warping to it, but it did not help yet. L1 loss on the parameters did help reducing the parameter count on the fancy elephant task. Here are a couple of failed runs:
https://users.renyi.hu/~daniel/elephants/
Reply on news.ycombinator.com