The better method for shrinking the SUSDB on the WID is to connect to the WID using SSMS and shrink it manually from there.
https://www.ajtek.ca/wsus/how-do-i-connect-to-the-windows-internal-database-wid/
Are you performing the proper WSUS maintenance including but not limited to running the Server Cleanup Wizard (SCW), declining superseded updates, running the SQL Indexing script, etc.?
https://www.ajtek.ca/wsus/how-to-setup-manage-and-maintain-wsus-part-8-wsus-server-maintenance/
@Pavel yannara Mirochnitchenko mentioned reconfiguring the SUP/WSUS instance (https://www.ajtek.ca/wsus/how-to-remove-wsus-completely-and-reinstall-it/) and you can do that, but it takes time and energy to continue to do that on a consistent basis if you continue to have problems. Performing the proper maintenance will prevent you having to reinstall (I'm using the same WSUS instance and database from 2014 to present day).