I think it's immoral, but it's much more immoral on the part of the actual cheating spouse.

That said, I wouldn't want to sleep with someone who is cheating on their partner, because I wouldn't want to have sex with someone who is callous like that.