1) Both of you win if you can get it into your brothers head that you are off limit without executing your threat. This is what we aim for.
2) Your brother wins by getting at you while you do not tell on him because you still want to protect him. (and you obviously lose from this situation.)
3) Both of you lose by if you execute your threat way too late.
The fourth option is the one you tried to avoid from the beginning - telling on your brother straight away and make him suffer the consequences. Play for the win-win obviously.

Frigid Onanoko sincerely apologize, these advices are based on ugly assumptions. However, even in the case that situation is much less severe, solution still very applicable. Be aggressive!

Gonna finally shut up now. whew... Bye now! and wishing you much strength!