it sounds like he is insecure of himself and doesn't understand why you are dating him, but it can be more complicated then that. I was like that for a while with my boyfriend, worrying about different things. The thing that worries about me and him though, is I don't feel that there is any real feeling between us. I think he likes me, but because it isn't vocalized I am not developing feelings for him. I'm not sure how he feels.
Probably the best thing to do is ask him why he has such a hard time trusting you. It could be more complicated then you think. But if you feel it is too much of a hassle then you shouldn't drag it out. It is better to end things at an appropriate time.