If it´s a problem or not depends on how YOU feel about it, not on what the others say :)
I think I´m pretty much like you in that way( except i´m a girl and you sound like a boy..but who knows?)..I´m a person with "weird/nerdy" likes so I find it very very hard to meet people who can really understand me. My real,true, close friends are so few, but it´s part of my own choice I believe. I´m really selective about the people I wanna spend my time with, so that leads me to be a loner most of the time. Anyway, I don´t believe those guys who have a million friends either..that´s impossible and highly probable they´re fake friends.
Maybe there are lots of people like us, but with our lack of ability to make friends or our tendencies of being alone, we just don´t meet,lol. It´s funny to find them all over the world through chat though.
If you feel ok with that, I don´t think you should worry. Greetings