This is such a great point. I've done something in my life that some people very much worry will lead to regret later. I don't think I will regret it. But I have thought about "what if they're right and this bad thing people warn me about really does happen". The way I see it, I am willing to take the risk. And if it doesn't work out the way I wanted, it will suck but it will not suck as badly as not pursuing what I wanted would have sucked.
There's a quote from a (problematic) movie that sticks with me: "If I make mistakes, they will be my mistakes, and no one else shall make them".
Sometimes I do make mistakes. But at least they are my mistakes. They came from my choices. Rather than from other people's choices for me.