I've also noticed this. I wouldn't be surprised if it took longer to refresh sources for folks who have an extensive list of rouge installer sources, as I do ...
It does take longer the more sources you have might not be a bad idea to go through your sources and delete ones that you might not use. I did this and it has increased my refresh speed tremendously.