Sex is not a part of the legal institution of marriage. Well, if there's a country where the law of the land says it's illegal to have extramarital sex, then I'm not aware of it.
People (especially women and their parents) used to tie sex into marriage much more back when sex meant children.. but it doesn't have to anymore which led to the erosion of that connection.
Anyway, sex is always going to mean different things to different people. Some people attach religious or spiritual meaning to it. For some, it's just part of their job. For others, it's their way of feeling superior. What does masturbation, which is essentially having sex with yourself, mean to you? If you look at virginity as just a social label, maybe you'll be able to let go.