That really is the point
A passport is a PHYSICAL identifier
They could care less how you feel
They want to know how tall you are, what colour your hair is (noone bothers to update this if they dye / bleach hair), what your sex is etc
Its like do you have a cock or not
Period
Not gee I kinda feel like a lesbian woman
The 'passport debate' is the same as the phoney 'washroom debate' — sorry about the euphemism, shoulda said 'toilet debate'. On one side are people knotted up about preserving their historic euphemisms, and on the other people insisting on taking those words at their face/behaviour/appearance-value.
Those who claim Men/Women, Male/Female designations are defined by physiology and nothing but physiology, would strengthen their case — and win at least one round — if they replaced those ambiguous signs and labels with their true specifics: 'Penises' and 'Vaginas'.
Wanna bet their oft-expressed convictions can't scrape up the courage to use the words?