For historical background on this part of the algorithm, called "experience replay", see this paper from Long-Ji Lin in 1992:

Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching

http://link.springer.com/article/10.1023/A:1022628806385

as well as his excellent 1993 PhD thesis:

Reinforcement Learning for Robots Using Neural Networks

http://www.dtic.mil/dtic/tr/fulltext/u2/a261434.pdf