If the number is reasonable you could just login and remove the unwanted connection manually.
If the agents have multiple connections (dual-homed), running the remote SCOM uninstall will only remove the connection settings (not the agent). As long as there is one lingering connection. You can easily test this on a few target systems to confirm.
I believe it may also be possible to run the setup command with switches to remove a connection from the command-line.
You might also look at helper or admin MPs from the community that might have tools around connection cleanup. Checkout Cookdown.