If we're not simultaneously acknowledging how the system also makes men feel worthless without sexual/romantic success, then yeah it comes off pretty misandrist.
Are we also forgiving men for saying they hate women because women's lib stripped them of the dignity of being a "provider" while women largely still expect them to provide? Not that I'm saying women's lib is wrong, it's certainly for the best in the long term, but upsetting the system affects all parties. Excusing resulting sexism on one side, but not the other, is itself sexist.
Why do we expect men to accept being hated for things beyond their control, instead of being treated as fellow victims of a broken system?