Because you are not and were not.
Yes, it is rude that they even reacted and said anything.
But your mommy doesn't put out your clothes on your bed and force you into wearing them. You make your own decisions on what you wear, how you cut your hair and groom yourself. In other words it is YOU that puts out your own image you desire to project. Even if you reply here with "I don't give a shit how I look, it is what is inside that counts", then that is what you are saying "I don't care".
Throwing out all your old Led Zeppelin t-shirts and cutting off that sooooooooo out of style "Old man wanting to pretend he is still hip/cool tied back ponytail", being clean shaven takes almost no effort and will drastically reduce prejudice you are subject to by the REALITY of any society.
Wal Mart, Costco and Winners sells sharp, decent, durable inexpensive clothing so there really is no excuse except that you WANT to look like a rough, dirty "I don't care" person. And that is your problem, not others my friend.