Part of the "self esteem" study is kind of flawed I think.

Someone with self esteem issues might initially feel bad after giving themselves a positive compliment, because their default belief about themselves is "I am not loveable, this compliment is false and fake and I'm pathetic for trying to cheer myself up like this". But if they keep doing it, it can end up affecting them in a positive way. Looking at how the self compliments affect someone over a longer period of time would provide more insight I think.

It also could be that people with low self esteem have lower mood overall than people with higher self esteem.

It's hard to know without looking at the study, which I tried to do but got a "aboutblank: blocked" result when I tried to click on the citation.