SSMS is based on the Visual Studio shell, and this is a core component, nothing which is optional. You cannot install SSMS without bringing the VS Shell, just like you cannot run a computer without a CPU.
If you want to avoid the VS shell, Azure Data Studio is your best option. (And it can be downloaded separately from SSMS.)