If you really care about him then let him know the things you don't like about him. If he really thinks of you as a friend he'll understand and stop being like that.

If you don't care about him then let him know why you don't want his company any more and say bye.

Hopefully he wont call you a bitch.