Every single time you push the clutch in and engage or disengage, you're putting wear on the throw-out bearing, clutch safety switch, and clutch disc. If you downshift almost every gear during regular driving, you're using the clutch that much more...simple huh ? I'm not saying it's going to outright ruin your car or drivetrain, however I am saying that for the average person, Downshifting is not needed 99% of the time.
Rev Match downshifting is quite a different story, due to the fact that it will put much less stress / wear on the syncros, and the engagement of the clutch disc to flywheel will be much smoother than a typical downshift. Even still, what's the point, so everyone can hear what your exhaust sounds like in each gear ? You downshift when driving in a spirited fashion, or snowy weather, or to maintain a desireable speed downhill, AKA when it is needed (like Apexing and tracking out of corners), not slowing down to traffic lights, that's just silly, really....
I'll reiterate that downshifting won't ruin your car and you likely won't even notice the wear for a long time, BUT if you don't Rev-Match properly, it IS putting undue stresses on drivetrain parts. Whether or not you notice the wear isn't really the arguement. I literally just called one of my old SR20 buddies, whom currently is a Mechanic at Skip Barber Racing School, and he insists that downshifting should only be done when needed, and not just to slow down, IE a situation where brakes will suffice.