How do I stop COM port from changing?

Question

I have an USB device that needs to be assigned to a specific COM port number (COM4). But sometimes when I plug in my device then Windows 10 assigns it to a different COM port number. I know that if I plug the cable to a different USB port on my computer then Windows will change the COM port number, but this is not the case here because the USB port being used is the same as always. This happens seemingly randomly. Is there any way to fix this? Without me having to change the port manually on device manager each time this occurs.