Since some international agreements that relates to passports don't allow someone to change their sex designation, a man who now looks like a woman (or had surgery and is now a woman) would have a physical appearance that contradicts their stated sex so the X would be a heads up to customs as to why.
And again, this is only the business of customs and the person involved and only that person will have to deal with difficulties in other countries. Other than not liking change or being afraid of gays, why the hell would anyone with a traditional gender care?
Let me clarify: I'd be fine with using "X" if it was a representation of a physical manifestation of sex. For instance, if you are transitioning. Though they'd need to educate people on the meaning of "X".What better way than an 'X' would you suggest, to inform people not to expect the holder's appearance to identify them as obviously male or or obviously female?
What I'm not okay with is using "X" because a person feels as though they are non-binary. What's important on a passport isn't what you feel, but the physical manifestation of what you are, for the purposes of identification.