EcoRNN: Efficient Computing of LSTM RNN on GPUs (2018) [pdf]
microarch.org
microarch.org
Earliest practical would presumably be the first round of GPUs with programmable shaders in ~2005 with the ATI R520 (X1000) and the nVidia G70 (7800GTX). Coming up on a decade later, ATI/AMD will surely commit to a cogent programming solution soon...