Have you tried to tune the settings in Google Map app to force the voice prompt via phone speaker instead of default Bluetooth audio?
I believe you can keep bluetooth connected to the car for phone calls while using phone speaker for the Google Map navigation.
