This happens with my phone as well. The only explanation I have is that, like with past jailbreaks, iOS 4 gets a little bogged down when jailbroken. I've noticed the fewer extensions you have running the faster it goes.
It's just the nature of having a modded device.
If you're running stock - then I really have no idea what the issue is.
I am jailbroken but, it also happened before I was jailbroken... I've seen a slew of posts on the apple forums discussingthis as well... Was just hoping someone may have found a solution on here