Clipping causes more heat in the voice coil and heat is the long-term enemy of both transducers and electronic components. A clipped waveform has more output which must be dissipated.
It's not a form of instantaneous failure, but over time is detrimental. That said, I tended to power-limit...