I checked with the internal team on this and got confirmation that currently this is the only way possible.
If personal desktop - then only way is to unregister the machine from the hostpool and add in again - this can be done by powershell but is the only way at present.