RDS collections "live" in your database, not on any servers that belong to your deployment. The only way to delete a collection is to use the RDS interfaces to delete it (or wreck your database, I suppose, although I don't recommend that). Removing a server from your deployment won't delete a collection or any of its VMs. Collections do not care which hosts operate which VMs.
If you remove a virtualization host from the collection and it hosts collection VMs, then the status for those VMs will become "Unknown" in the RDS displays because your deployment will no longer look on that host for VMs. Usually you can make things right just by moving the affected VM(s) to a host that still belongs to your deployment.
To do this with the least pain, move collection VMs off of the host that you want to retire and onto hosts that will remain in the deployment. If your virtualization hosts are clustered, then you need to make sure that a VM cannot automatically migrate back to that node. Usually with a retirement, you evict the node, but you can also pause it. Then just use Server Manager or the RDS cmdlets to remove the ailing virtualization host from the deployment.