I tested many different batch sizes for the LSTM, so I am pretty confident it's not the reason.