The training is most certainly supervised. The fact that people commonly draw a distinction between standard supervised learning and reinforcement learning isn't really relevant here (the distinction between a single datum and a sequence of observations), because the fact is the label is the score. If you read their DQN algorithm (or try to code an implementation, or look at any reinforcement learning algorithm, really) you would see that.

https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf: In Algorithm 1, see variables r and y_j.

Addendum: My broader implication is that if you misunderstand such a basic aspect of machine learning (and reinforcement learning), it makes me question whether we should lend credibility to your predictions about the field.

Edited to address reply.