One thing that popped out is that you might want to try to define dependencies at the workspace level and then in the sub-crates point to those instead.
Otherwise you are correct, your setup is already very granular. Furthermore, I have no clue about why it takes so long on your end :/ . I am running on Linux, so I wonder if there's something in the MacOs stack that messes things up and so you end up recompiling dependencies over and over.
Another thing I noticed in my setup is that one of the biggest offenders was sqlx if the macro feature was enabled since the queries were ran at each incremental compilation.