I think that's part of it, but there's another factor. My guess is that with so many human bodies to fill with souls, due to overpopulation, souls are being rushed back into bodies too quickly, so they still dimly remember their former body. I've heard transgender people talk about how their bodies just feel wrong to them; if you'd just spent eighty years in a woman's body and then got reborn as a man, that's by no means an implausible reaction.