Hello Leon,
I am Jaspreet Singh.
Sadly this is not an issue with the headset or your system but a limitation of Bluetooth itself. In the talking or headset mode the audio bandwidth available for receiving audio is reduced to provide it to the audio being transferred from the mic hence the reduction in quality and why it improves when you turn off mic.