Yes, bellman equation fundamental idea for all RL algorithms (i.e. updating table of values), but RL is also much more fragile than supervised learning methods, thus to ensure stability, modern algorithms have complex mathematical tools to fix that. I wouldn't say it's the math notations that's scary, rather the concept behind modern algorithms require higher mathematical concepts. For example, Max Entropy algorithm for inverse RL, it's essential to know the concepts of Shannon information entropy to understand why it works.