So, while I agree that the restroom stall sign won't stop a determined rapist, it's not true that it's not illegal for a man to enter the women's restroom. The legal situation here is variable, but there are precedents for it being trespassing (entering an area you are not authorized to enter).

But I did a bit of research. In the City of Portland, it is "unlawful for any male person to enter a restroom marked "Women." And vice versa. The law exempts children with a parent/guardian...they should use the restroom determined by the parent's sex. It does not exempt disabled people who's caregiver is of a different gender.

It's illegal to use the other bathroom in Houston without permission from the business owner.

Most school systems require that children use the "appropriate" restroom (sometimes based on their genitalia, and there have been instances of teachers checking).

Park bylaws commonly restrict bathroom use.

Assuming it's legal to enter a bathroom designated for the opposite sex is...well, you can't assume it.