Sounds like a solid plan. No reason to incur switching costs if things are working!
I think you're misunderstanding what "removing pandas" means.
You can still compute on DataFrames, it'll just be with something other than Pandas itself, probably DuckDB, Polars, or DataFusion.
So, the bridge was there, is there, and isn't going anywhere.