In electronics, as usual in situations where a cricuit is misbehaving and we don't understand why, a solution that generally works is to add a decoupling cap between +5V and GND.
In your case, I would put at least 220uF; that would ensure two things:
- the voltage used to control the servo never drops
- the voltage brought to Arduino by the USB does not drop either (which would have very bad impact on the Arduino, probably resetting it)
Electrolytic caps are a must-have for every electronics hobbyist (and professional as well, of course).
You can read more about decoupling capacitors on Wikipedia.