I mean, going way too slow can be a problem...I remember when I was learning to straight shave, I would take like 20+ minutes per pass and my lather would dry out on my face! Short of that, going as slow as you need to is probably the best approach...getting blade angle and pressure right is far more important than the speed of the stroke.