I feel like you maybe didn't have enough access to good alternatives to meat and dairy.
I'm a vegan and it's not difficult. I don't crave meat, and for the dairy things I want, I can get vegan alternatives that are delicious. Chocolate ice cream? Oatley. Chocolate? Lindt oat chocolate with salted caramel. Mocha frappuchino? Just ask Starbucks to make it with almond milk and no whip. Cheese? My local co-op makes delicious artisanal vegan cheese out of cashew. Yogurt? There are amazing coconut or almond versions available at Safeway.

I applaud you for mostly eating vegetarian but I would suggest you try some of the vegan alternatives to the things you crave. You may end up liking them as much.