Reinforcement learning is supervised learning on optimized data
bair.berkeley.edu
bair.berkeley.edu
1. learning a policy that imitates your own behavior on prior experience, which is a trivial supervised learning problem
2. learning how to weight the importance of prior experiences (learning a data distribution), for which the authors have derived a lower bound
Given a pool of experience, this seems like a fantastic off-policy method to optimize arbitrary reward functions. The main shortcomings I see with this method is that it still does not lead to any significant insights into how to collect new data online, which is a major open problem in RL.
This is a blog post. It cites three of the authors' papers that each contain empirical results. The abstract of the first ends:
"We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms in several benchmark tasks."
It's interesting that their choice of current algorithms includes PPO but not e.g. Deepmind's Rainbow agent that achieved state of the art performance on many measures: https://arxiv.org/abs/1710.02298
Gradient descents are a special case of associative learning rules assuming all data points the same importance.
A type of associative learning rules is Hebbian learning rule.
In the very fundamental we only need associative learning. Of course, for practical application we need diverse tools with different conceptual frameworks to choose for commercial: human resource or cost performance.
No. Gradient descent is an optimization method that has nothing to do with learning. It is used to optimize parameterized functions that are said to be "learning", but it's not the only approach. It is also trivially easy, and not uncommon, to use different weights for different data points.
Can you clarify what you mean with associative learning?