Seconded. I lost my virginity in a similar fuck-buddy situation and, although we consistently hung out (read: watched a movie, got some dinner, talked, and then fucked), he never would talk about it with me, easily got frustrated by my relative inexperience, and was, overall, a pretty borderline guy; funny to have at parties, interesting to talk to on an individual basis, but hard to foresee being long-term involved with, well, anyone.
My next boyfriend ended up being a virgin and, granted, the guy has a pretty liberal upbringing. He, however, was a virgin and, rather than being turned off by my prior relations, he felt good about me having some idea what I was doing.
Granted you're clean and safe, most people really don't care. But fuck buddy relationships like the one you were in? Unhealthy. Breeds jealousy. And you seem like a pretty sensible lady, so find yourself a sensible man that appreciates you and gives you the attention you deserve.