No, I think you reacted correctly. Not that I think he necessarily intended you harm -- but he recognized that you were alarmed by his closeness and yet didn't back off, which is rude at the very least.