In the training phase the network learns to predict the next note from the past few beats. As training data it uses a bunch of text formatted midi files. In generative mode, they just feed back the generated notes into the network at each step. The same approach can be used to generate DeepDrumpf[1] or any kind of time series.
[1] https://twitter.com/deepdrumpf
Funny, this is DeepDrumpf bad mouthing Andrej Karpathy, the inventor of this algorithm (char-rnn):