I mean I personally think it's a good idea to try and find alternative ways of satisfying your desires that aren't breaking your ethical code.
It doesn't have to be either-or. It doesn't have to feel like a sacrifice. You can have an equal amount of pleasure in a different form if you put a bit of effort into it.
When the choice is down to "will a sentient animal die horribly for my momentary pleasure or can I put a bit extra effort into finding an alternative means to satisfy my desires and still enjoy life to the fullest"? I think it's good to try your best to find those alternatives that can allow you to live happily with your ethical choices.
Being vegan or vegetarian does not have to be a sacrifice. And it's not fair to compare it to polyamory because in polyamory everyone consents. Whereas in meat eating the animal does not consent to be murdered. In dairy the cow does not consent to being raped (artificially inseminated) and the male calves do not consent to be slaughtered at a few weeks old because they're "superfluous".
I heard a good line that speaks to me: "being vegan is not a sacrifice because I'm giving up something that was not mine to take".
It's also not something you need to martyr yourself over. I eat just as deliciously as my meat eating and dairy eating friends. I am also a lazy cook so I don't even work very hard.
My average daily meal schedule:
Breakfast: frozen berries and vegan yogurt with nut butter for breakfast, with oat milk caramel flavored iced coffee from Califia Farms
Snack: spicy hummus with carrot or celery, pomegranate seeds, cherries or other seasonal fruits
Lunch: in summer, a massive salad with mixed leaves, sundried tomatoes, olives, cherry tomatoes, cucumber and vegan cheese with balsamic dressing and sourdough bread. In winter either vegetable soup with bread or vegan cheese and tomato toasties.
Dinner: usually either Indian or Thai food. If Indian it's channa Masala or dal with rice and roti, with vegan vegetable samosas. If Thai it's usually a green curry with peanut sauce added.
Dessert (I don't eat this every day but often when I have munchies): vegan lindt chocolate or Oatley ice cream
FYI I'm NOT saying you're a bad person or a monster for eating meat sometimes. I'm just suggesting that maybe it's a false dichotomy between satisfying your desires for delicious food and living true to your beliefs (since you already said you feel guilty when you eat meat).