I think the ones who look away do it because they don't know the game as well... so being stared at creeps them out.
An attractive woman may think 'I'm hot, you know it, I know it, I worked extra hard at it today - so I'm glad you're noticing, yay me!'... Someone who doesn't find herself...