Should I have him be a part of the wedding?

No one has the right to abuse you and that includes family. The most important thing is your feelings about your brother and you have clearly stated 'I really don't feel comfortable around him', as far as I am concerned that is sufficient reason NOT to have him at your wedding. He is already proving to be difficult, so my advice is to tell him you don't want him there and be honest about why!