It's true, if you think about it, even Neurotypical people are prone to social anxiety. So if they pick up that someone is uncomfortable around them, they're likely to assume THEY are the one making them uncomfortable and want to stay away because they don't want to bother you. Whereas if they know you're autistic they are more likely to not get in their own head like "omg this person instantly dislikes me". Which is what you are also thinking.

It's a big misunderstanding, a lot of the time.