I don't agree with this, as a bisexual woman myself. Being bi doesn't mean I have to have sex with people of all genders. It just means that I *can* feel attraction to people who aren't just men. It doesn't mean that I am not able to be happily monogamous. And it doesn't mean that I "need" to have sex with women to be happy. Women are people, men are people, I am not attracted to just all people of all genders but to specific humans.
So yeah, me finding a specific woman attractive and wanting to sleep with her, is equivalent to your husband finding a specific woman attractive and wanting to sleep with her.
Because both of you are being attracted to specific people. The fact your pool of "people" includes both women and men, and his only includes women, doesn't make that any less true.
I had a convo with my husband recently because we are poly, but he was not particularly keen on me having sex with another man, but was fine with me having sex with another woman. But he expected me to be ok with him having sex with another woman. His rationale was the inverse of yours, that because he was straight and I was bi, he couldn't have sex with another man, but I could have sex with another woman, so for him to be poly he would have to be with another woman or nobody. And because we are poly not open, if I was with another man he couldn't find him attractive, but I could find another woman attractive and "share" her.
We ended up discussing it and I made the case that if he found another woman, there's no guarantee that I would be attracted to her because just because I'm bi, it doesn't mean I'm guaranteed to be attracted to any specific woman, it's on a case by case basis. So we ended up agreeing to be open to me being with any gender and to not expect to find each other's partners attractive, but that we would expect to at least be friends with the other person's partner.
Not that we're even doing it at the moment, but I think our agreement is more fair.