According to Recovering from an Invalid-Device Error, It's enough to
Release the IAudioClient interface and any other WASAPI interfaces that it has acquired on the device.
There could exist any other WASAPI interfaces that it has acquired on the device and not released. See bool CWASAPIRenderer::HandleStreamSwitchEvent() for an example.
It's better to show a minimal, reproducible sample without private information.