You'll have to make your own judgment call. Not seeing each other face to face except for summer isn't going to work out, no matter how much you like him.

Do you have a means of visiting him? Perhaps you can drive out every other weekend and spend the whole day with him (or sleep over, if you're okay with it.

Remember to keep lines of communication open. You should try to contact him at least as often as he contacts you.