I just thought it was me. My first Mazda. Brought it in and they reflashed/updated the software version (supposedly) but it still does the contact reload each Bluetooth connect.
I finally gave up and just connect through the USB cable.
I turned it off the auto download contacts last week. In the few times I've driven since then it seems slightly quicker to boot and load up.....or just imagining it. As previously said, one less thing to bog the startup can only help.