Well then you know you're not compatible, or you adjust together?
When I joined what was initially a polyamorous relationship with my partner, they practiced "polyfidelity" where they wanted everyone to date each other. I was initially open to it but said I couldn't force myself to have feelings for people, and it would have to occur organically. And so the agreement was adjusted. But if they hadn't wanted to adjust, that wouldn't have been wrong, just as I wouldn't have been wrong to stop dating my partner because I didn't want to date his other partners.