It's commonly known that people are unable to perceive changes when they happen underneath 10ms after their action. This responsiveness will result in an experience that has recently mostly been described as "snappy". It's noticeable but for users it's hard to put a name on it.
So if you want perfection, take about 15ms of delay. If you want really good, take 100 ms of delay. 100ms is 50ms on average, and will certainly pass for people.
The application and the expected response time is vital too. A sliding door or elevator is given very large tolerance (as the physical object will always take a lot more time) whereas ticket vending machine interfaces are not given any time at all.
The upper limit for polling would be around 1500ms. Around there people will always notice it is slow.
This data is purely personal experience as a gamer and programmer. YMMV and remember that just trying it yourself is the best way to find out how it feels. The only "scientific" answer is the <10 milliseconds, beyond that it's about the ability to perceive the delay (which varies per person and moment) and the tolerance of the user.
As a side note, you can try fluctuating the delays in order to conserve battery or CPU time when the interface is not being used. The user-action, the faster the polling. When the application is doing it's thing, poll very slowly. Better to poll when it matters!