Should i have given him an answer?

You seem like a very reasonable person, and a person like this in your life will just cause drama. You should never sacrifice your happiness. It seems like you deserve more than this guy. Keep him as a friend and let him know that you will be there for him but find another guy that will treat you with the amount of respect that you deserve, because you do deserve it. Hope I helped C: