Yes, I think it's perfectly fair. You were spooked, and likely would have been regardless of who suddenly fell into step behind you. That he realized this, commented on it, and did nothing normal to ease that (namely, letting distance come between you, and to stop talking), is, well, weird. And it would have contributed to my own sense of menace. It's possible that he thought his chatter would calm you. Perhaps that was his goal so that whatever underlying plan he had could be carried out. Perhaps he really did just mean to soothe you. But once I've been spooked, the last thing to calm me is soft words from the stranger who caused it.

I agree with you that no, prejudice didn't contribute to your sense of danger. YOu didn't feel in danger until someone started keeping pace with you. That it was one of the dealer on the street is inconsequential.