tl;dr: Break up with him immediately. If you refuse to do that, you deserve everything that happens to you.

I've learned from listening to Dr. Drew in the evenings that people with abuse in their past have "freeze reactions". This is why you get stories like "he raped me at a party in front of everyone", where one word from her and everyone at the party would be prying this guy off of her. If you ask the guy, he thinks she consented because she didn't object.

Unfortunately, the guy has to have some clear indication that you want to stop, AND sufficient time to actually stop. Because sometimes "no" means "not there, yet" or other such ambiguity, and he might not be paying attention.

If the guy has that clear indication that you want to stop, and he does stop but takes too long, that's not rape, but it's also not good if it makes you feel uncomfortable.

But if you can't work out a rhythm after a month or so of this, then clearly there's something dreadfully wrong. It doesn't matter which one of you isn't adapting, because the solution in either case is the same: break off the relationship. Completely. Block his phone number, don't answer the door when he knocks. If he approaches you in public, enlist the help of friends to keep him away from you. And if he stalks, then call the police.

If you do not break up with him, you're expressing consent by allowing the status quo to continue.