What's wrong, I think, is expecting that I should be able to put myself into a position where my race doesn't ever matter because I am behaving correctly on racial issues. People of color don't ever get to be in the position where their race doesn't matter, and it's a reflection of white privilege to believe that if I "follow the rules," I should be able to be there myself.

It is, but I'm not sure the solution is to give up demanding it for myself, rather than to start demanding it for others as well, or at least to get out of their way as much as possible so that they can demand it for themselves.

I find it difficult to talk about these issues in the abstract, though; in real life, my reaction would be heavily dependent on why the other person in the conversation had chosen to raise this issue (rather than some other topic) with me (rather than someone else). Venting calls for a different response than a request for a specific action from me, which calls for a different response than a personal accusation, which calls for a different response than generalised aggression based on my perceived ethnicity.