Bearing in mind that I don't know either of you in real life and am only going off what you've written (so feel free to ignore), It could be that he doesn't feel free to open up to you yet because:
- the fact he's begged you for 4 years to be monogamous and you're only now trying it might imply some resentment on your part about being monogamous when you don't want to be, and so he doesn't feel able to trust that this monogamy will be long term, so he's holding himself distant to avoid being hurt if this fails
- It makes me wonder were there times that he reached out for comfort and reassurance from you over those 4 years that you ignored (or didn't realise)? Cause what you write about him depicts someone with a lot of pain that they don't feel safe in sharing because they think they'll be shamed. I don't see this as being entirely his fault for the issues in the relationship.