I have been through this path of trying to figure out why pandas Groupby is slow. It is a big bottleneck when building packages on top of pandas.
I think at its core, the groupby issue is a really big problem, and am devoting most of this year to working on it. So if you ever want to pair to work on pandas / pandas wrapping libraries send me an email (link in profile)!