This could be just me but my observation is that men have more sensitive nipples than women. We are also less able to get into aggressive nipple play as our pain threshold is lower. I suspect why the many men have said they don't like it was because when it was tried on them the lady went too hard for them. This takes experimenting and a partner you can trust.
Many women figure that guys are rough and tumble so thats how they caress us (our penis is built for rough) but there are some spots that are more delicate than that of a womans body. This however is my opinion, you'll have to experiement to verify this.