I also think some IE updates are left behind after the uninstall. But they are hard to find since they only say "Update" and not "Update for Internet Explorer" after installed. Going through my update checklist I made I can only see an IE8 compatibility update and an IE10 update. But it appears I have to try this over again since my svchost.exe is at 54.024K after 3 days with IE11 installed.
Edit:
After 4 days uptime it's at 303,612K. I have no idea what happened to it in the last 24 hours. It just started jumping up really fast. I haven't done anything different. I really wish we could figure this problem out.
Edit 2:
Disabling and re-enabling the network adapter still works at least. It dropped instantly to 51,004K.