Eight out of 10 men say they notice changes when they don’t exercise regularly. Those can be physical, and they can feel more stress, become more irritable.