Reinforcement Learning: An Introduction, Second Edition
incompleteideas.net
incompleteideas.net
One of the (many) things I still don't understand about RL even after having tried to read about it is how the loss function of Q Networks is computed if you don't have the target value. I understand that you are trying to predict the "value" of a pair (state,action), and I know that there is a formula for updating the weights based on the difference between the expected value of the next state and the maximum predicted value available plus the reward when you actually get to the state (probably I'm messing up here), but then doesn't that mean that the initialization is extremely relevant? As in, how will the rewards ever be superior to the random weights already present in the network?
The basic (non-deep) Q-Learning approach, is to have a big table of every state-action and it's expected reward. The formula you've seen (the Bellman equation) is for updating these.
Using big tables to store Q values sucks though, so instead of using a table, you use a neural network.
But the bellman equation works just the same with infinite horizon problem. The weights converges all simultaneously to make the network approximation solve the equation.
A great resource to understand RL is openAI's spinning up.