Lets be clear, stare to me means "continue to look:".
So when someone says that I 'shouldn't' stare at them, they are saying that it is somehow wrong that I do so.
Now, if a person , say, has a physical deformity, or even is exceptionally tall, short, etc, staring can make that person feel uncomfortable. They can't help why they look the way they do.
But when a person UNLIKE what i just mentioned, voluntarily acts significantly outside of those norms, (and clothing choice is an act), it is somewhat unusual. Lets say someone walked around with a hammer glued to his head. I perhaps might stare awhile, and make no apologies for doing so. it is interesting, unusual. We focus on the unusual when we come across it. It is no different than sitting on a bench, and observing a car accident.
Now, couple that unusualness with a case where the deviation from the norm ALSO clearly stimulates the sexual drive, and in fact to some extent SUPPLIES a sexual experience - after all, men PAY to see women in various states of undress at strip clubs! - then you are gong to get quite a few men who want to enjoy that stimulation more.
I believe most people know this.