This is very true. My partner had a psychotic break due to magic mushrooms interacting with his existing mental illnesses. It was life ruining. An innocent person died due to my partner's psychedelic triggered delusion.
On the other hand I had an experience on psychedelics that cured my eating disorder and gave me the sense of being seamlessly one with the divine.
They can be heaven or hell. If you don't want to take the risk, I completely understand and I think it's a wise choice for you to make. You know yourself better than anyone else.
I wish more people were this mindful and careful with psychedelics, they're definitely NOT for everyone.