
This can largely vary, but first off make sure you're running a farm that has been patched within the last few month's of Public Updates -- there were some glaring issues until recently with WSP deployment.
Retraction takes time depending on what the code as part of the retraction process needs to do, i.e. code that has to also delete a custom service application or instance is going to take longer than one that doesn't. The other thing is how many servers in the farm. And then we get down to how the farm members and SQL server are performing.
Unfortunately via your post there isn't enough information to help you determine why it is taking so long but diagnosing that is likely something that can only be done by working directly with your farm, I don't think this is going to be an issue that ULS will tell us about right away.