They felt it was an invasion of their territory, humans and other animals, especially males are very territorial. They felt that some other man was taking their woman away. It really has little to do with sex, any physical contact would probably bring on such a reaction. The only difference is that they probably would not call her a whore for holding his hand instead of sleeping with him but they would still be jealous and angry.