Is he gay? If so, you may have nothing to worry about, atleast from him.
Does he talk to your girlfriend ALONE? Meaning no other party?
If he does. It might be better to look into this a bit more.
If it's just the threeway, then you may be overreacting just a tad. Although regardless, I do see why you would get uneasy about the whole thing.