I found some info that might be helpful for anyone experiencing this problem. If the external device's (in my case an external ssd) removal policy is configured for quick removal, which for me says it's the default, then you don't have to use the Safely Remove Hardware button. It's configured such that unplugging the device without clicking that button will not corrupt data (unless you pulled it out during a file transfer I would assume). You can check the policy in the Disk Device Properties > Policies tab. Right click drive in explorer > Click Properties > Hardware Tab > Select problem drive in "All disk drives" list and click Properties > Click "Change Settings" > Policies tab.
I'm guessing this is more or less a UI bug on Microsoft's part. Since it doesn't need to be ejected, it's probably returning/utilizing invalid or malformed/unhandled data to the Safely Remove Hardware command/process and it's just generically returning an error message about it since it doesn't know how to handle it. Something like that. If it's set to Quick Removal, you should be able to just unplug it when you're not using it.