Metamine was an experimental system that combined declarative as well as the normal imperative code. You had a "magical equals" that kept the left side updated as the conditions on the right changed (like a formula in excel). It allowed for some very interesting things. It got pulled from the internet, though.
I'm not super familiar but it looked cool when I read about it a few years ago
Also, differentiable programming is interesting and getting lots of attention (jax, zygote, enzyme, kornia etc). I don't know if there is a "killer app" outside just training neural networks yet, but it's worth watching and e.g. kornia is doing some cools stuff
in my mind the work on extending our reach into quantum computing models is the only “cutting edge” stuff happening right now.
the is one of the most mature areas of computer science, there isn’t a whole lot of cutting happening imo, the blade is plenty sharp.
Memory management also may be fertile ground.