There is more to evolutionary fitness than reproduction, though that is chief among the desired traits. The gathering / providing of supplies and wellbeing of the home are also up there.

Not to mention, wrt reproduction we need all the women we can get. We need only a few men. One male of a species can keep an arkload of women pregnant at a time.