Very true. I get this also when I tell people I'm vegan. They automatically assume I'm judging them. But I'm not. But they take it as such, for this exact reason.

I also don't drink, for the same reason as you, but I do use weed. :P So I usually tell people I'm California Sober lol.