Explain him that you will not accept that things continue like this. Be very clear on that. Either he accepts to do stuff with you, or the relationship has no meaning.
You can start by suggesting things to do. He can only refuse a suggestion if he can himself suggest something to do outside of your place. If he has no ideas, he has to come along.
It's not foolproof,... but if the situation is not too bad, he will start cooperating.