My take on it is this: he's a bit of a dickhead. Your friends split because they don't want to hang out with him. More importantly, however, you find yourself constantly annoyed by this guy. My advice is to tell him to go fuck himself. His attitude is a pretty big sign that he's too inconsiderate and self-centered to show respect for your friends, if only out of respect for their friendly connection to you.

I found myself in the same position up until about a month ago when a girl I'd known (and was attracted to) since we were little kids finally just got on my nerves to the point where I've flatout stopped interacting with her. I don't even care to tell her why. Basically, she constantly talked crap to me about other people. No one else I knew liked her.

Anyway, my advice is to toss this fish back into the sea. Out of curiosity, are you from Japan? Kind of a dumb question since your name has Japanese characters in it, but the Internet is the worst place to assume you know something.