What does sex even mean? Well, there is more than one way to look at it. On one end, you have the view that our sex drive is nature's way of keeping the human race alive, and having sex for pleasure doesn't hurt anyone if you do it safely. On the completely opposite end, you have religions which teach that sex is a very spiritual thing, and thus should only occur in married couples.
The idea behind the latter is that having sex creates an undeniable spiritual bond between the two members, and is dangerous when done with somebody with whom you don't have a strong relationship. Whether you believe this or not, the fact is that sex is a fun activity, but there's no escaping the connection to "love" (whatever that mysterious word means) that sex has. Because of this love-connection, I personally believe that "fuckbuddies" are an inherently bad idea.
So, call me old-fashioned, but I think losing your virginity is a big deal. You shouldn't have given it away to such an asshole. But, don't fear. It's not like it will ruin your life. We don't live in the dark ages anymore.