It can be done. Some people remove their A/C compressor because they have modified their exhaust and are running a turbo while others want to remove for weight just as you want to. Use the search button I have seen it on some forums because someone already figured out what belt size you will need once the compressor is removed and the pulley set-up.
I could never remove mine even though I hardly use it. I do tend to need mine for removing the fog on the windows as I live in Florida where the constant rain makes it very muggy and causes my windows to fog up.