The people you are describing are not what a most of the people have issues with. People are born with the scenario you describe. It isn't a lot of people, but it does happen. Typically the parents chose the path of the child. Then when the person gets older they can figure out what to do.
There are far more people who for some reason decide they were born the wrong gender so they do what they can to change that. Either by taking medicine, mutilating genitals, or just swapping clothes. Its obviously a mental thing, as they are born body wise the other gender. The question is how do we handle that as a society. Of course they can control what they do with their own lives, but when it impacts others what should be done?
Letting a former grown man use the bathroom with female kids just seems a little weird. Letting a dude who thinks he's a female use the mends bathroom? Still a little weird. Create transgender bathrooms? Seems weird and never gonna happen for the majority of places. Letting a former guy compete/fight females in sports is also weird. So who knows that do to.
My one hard stop is giving kids hormones one way or another to change them. Just because a kid plays with barbies or wants to wear a dress doesn't mean you should inject them with medicine to alter them.