Not surprised, we can come up with probably thousands of similar scenarios when some popular free tool ain't the smoothest to use for 'when I do this but want also that and while doing something else that isn't so well integrated into the flow something happens/doesn't happen but should' situation.
Maybe take a laptop and a mouse next time, 100x more efficient to use. Or use it in browser in tabs, so you can have 20 different locations/views and switch between them.
Using Maps while driving for anything other than A-B navigation is generally poor. It seems designed by people who are accustomed to every destination having plentiful dedicated parking. If you want to go to B, then find local parking, you need to invoke dark voodoo.
The worst part is that the Apple & Google duopoly in maps supported by Android Auto and Apple Carplay means there's no way for a superior alternative to succeed in the market. Google even contributed to this by buying Waze. So Maps only needs to be just good enough.
What does Android Auto and Apple Carplay have to do with it? Aren't both of them available for third party apps to use?