You can’t think of any circumstances where a man claiming to be a woman to enter what is generally a space reserved for the opposite gender with malicious intent wouldn’t be concerning?
Surely basically any changing room or bathroom in the country meets this criteria?
The examples you give are bad - in those spaces you are judged on how you look, not your legal gender. Many men and women have been questioned in changing rooms / bathrooms just because of how they look, even if they’re cisgender.
So I can’t really see how changing your legal gender freely is going to cause issues. There is nobody stopping a man going into a womens changing room or bathroom already.