He's a fool. Plain and simple. Muslim or not you have the right to live and work in a safe and amicable environment which is free of harassment and sexual misconduct.
Don't go around in circle trying to be nice while preserving your dignity. Make it explicitly clear you will escalate the issue to the necessary steps in order to be left alone, whether that means your boss or a restraining order from local authorities.