Since the authors appear to be listening on here, I have a question about the method.

From my experience, RNN weights and the recurrent weights Tf in an LSTM tend to look more like (I + low_rank) rather than low_rank. To be more specific, I gather that with your F-LSTM you do:

  T1 = W1 * input
  T2 = W2 * T1
  output = T2
  so
  output = W2 * W1 * input

Where W1 and W2 are "factorized by design".

However, it seems like the recurrent weights (those for f in your paper) should look more like

  T1 = W1 * input
  T2 = W2 * T1
  output = input + T2
  so
  output = (I + W2 * W1) * input

That way, you are imposing the simplification that the Tf ~= (I + low_rank) instead of Tf ~= low_rank. Have you considered this?