No, you can't blame only on consumers. It's Mazda's problem that they can't keep the customers after they have tried Mazda, hence very low customer's loyalty or customer's retention rate. Automakers average 53-percent loyalty, but Mazdas brand loyalty was just 39 percent in 2016. So why so many consumers who have owned Mazda and then decided not to buy another one? Quality and reliability issues? Or something else? (uhm)
I'd wager it's likely multi faceted.
There's probably some left over negative sentiment from those that bought during the less than ideal 'Ford era' for starters.
Then there's that they're a relatively small company and have somewhat limited offerings for buyers to 'move up' to. If you've outgrown your current Mazda and need 'more'...there's simply limited options there.