I think you may have hit the nail on the head, regarding being concerned about being punched back. That would be it for me, anyway. I don't trust the male ego enough to put myself in that position.
Though, this is coming from someone who has kicked a dude in the balls because he paid me haha.
But even then, it took a lot of time before I would do it. Yes, I was a Domme, but that doesn't mean I'm necessarily a sadist. Also, the concern was definitely there, that he would decide he really didnt want that and turn violent.