Ok, I way overcomplicated this:
- Reinstalled Azure AD Connect.
- Synced all items again to get things to where they were.
- Created a blank OU
- Pointed sync to only sync that one OU.
- The rest vanished on sync.
Much cleaner and efficient. Should have known to do that in the first place.