I'm a 23-year-old senior in college, and while I wish I could tell you that it gets better, it really doesn't.
On a small campus, the population can tend toward homogeneity -- good if you fit in with the majority, bad if you don't. For example, I attend a Catholic college, and many of the students are pretty hard-core about their faith. I'm an atheist. Generally, things go about as amoothly as you'd expect. I've also known a few GLBT students, and they haven't had the best of times, either.
On a larger campus, the population tends to be more diverse, but your likelihood of encountering any one student drops substantially. Many of the big state universities are like cities unto themselves -- you can spend your entire college career flitting about like a social butterfly and still come into contact with only a small fraction of the total student body.
Either way, if you feel you don't fit in now, it's not likely to get better. Best thing is to either try and make friends off-campus or try and accomodate your collegiate peers. I've chosen the second route, even though it means that I have to be less than completely honest about my life and who I am.