With 3D the cooler still goes on top, 2.5D vs 3D is more about how the signaling and interconnect is structured. When the interconnects are still mainly planar it will be 2.5D when most of the interconnects are throughsilcon it’s called 3D.
Cooling 3D is more challenging because you need to wick the heat through the stack. It’s usually done by embedding additional metal layers for heat wicking but there is also a lot of research these days into micro channels for in-die liquid cooling.