I think you should make things clearer to him. Not just letting him realize that you are offended. Tell him. Say something. I do that most of the time, when I'm offended.