If you start messing with the weights on the crankshaft, you inherently throw it off balance. This means that if you put a lighter crank pulley, you effectively compromise the balance. Also, the stock pulley has a harmonic balancer that dampens vibrations. This also applies to forged rods and pistons. The result would be an unsteady or rough idle, with your engine ultimately failing somewhere down the road because your unbalanced crank will ware out your bottom end bearings.
If you do anything that may throw the crank balance off, you should get your crankshaft re-balanced (~$150-250), this ensures that everything is rotating in harmony.
That makes sense. I guess I just can't tell the difference since I changed so much when I installed mine lol.