Bonus points for the first to reimplement this in PyTorch or Tensorflow.
Seriously, that's very cool. Well done.
I've tried to add neural time warping to it, but it did not help yet. L1 loss on the parameters did help reducing the parameter count on the fancy elephant task. Here are a couple of failed runs: