when this case happens I think you should probably blame the author more for not giving the males more diverse personalities or even hinting that the other males in those universes can be decent people too rather than focusing your hatred on the protag, and for creating a world that facilitates "mc gets all the girls because he's nice."
think about it, if you live in a world where everyone's a shitty/abnormal person and is out to get you/take advantage of you, and there only existed one person who tries to look out for you and generally has good intentions behind spending time with you, wouldn't you gravitate towards that person too?
to me it seems like what those girls are doing is perfectly logical. if they lived in a world where everyone was like Rito but they fell for Rito even though he would be indistinguishable from every other boy, then it wouldn't make the slightest bit of sense, but in Rito's world, that's not the case. the point is that even though Rito as a person is very ordinary for our real world standards, in his world, he's extraordinary and he stands out, and that's why all those girls are focused on him.
this sort of thinking seems like it would really make sense with 4chan's perspective on girls in particular: that they only ever fall for the exotic/extraordinary people who set themselves apart from everybody else and that they're unable to lower their standards and be satisfied with an ordinary guy. in Rito's world the "ordinary, mundane guy" would be the shameless perverts like Rito's friend or the guys in Momo's fanclub.
this is why I have a hard time hating this kind of harem protag because I see that, generally, what happens to him is a consequence of the kind of world the author created for his characters to live in, and that by himself, he has qualities that I would like in a person if I met them IRL.