Well, I monitored the memory usage for an hour or so and it stayed at around 75MB with Norton Internet Security running. Given that, I don't think that uninstalling it would have made it easy to spot any difference in performance unless I left myself without any antivirus for a couple of days to see if it stopped the problem. I did try running a scan and runtime broker did not move from 75mb.
Later on today it started to go up again though and is currently at just over 1.1GB.
I upgraded my Windows 7 installation using the Windows 8 Pro Upgrade disk that went on sale on the 26th October, not using the RTM version either. Whilst I've been typing this I've decided to be a daredevil and uninstall Norton. I will observe runtime broker tomorrow and see if it makes any difference.