Hi my-sukiyaki,
Thank you for reaching to Microsoft Community.
When you leave the device idle for a few minutes, does it eventually goes to its Sleep mode? Is the power supply connected during this time? If not, can you test it if the same issue happens if the power supply is connected?
Please run these steps first on your Surface Pro 4 then check if issue persist:
1: Run Surface Diagnostic Toolkit and check Windows Update
2: Update drivers and firmware
Check what is the current OS Build of your Windows version by going to
Settings > System > About.
Go to Download driver and firmware for Surface, choose Surface Pro 4 then click
Download on the next page.
Choose the .msi file that matches your current OS Build and click
Next.
After downloading, run and install the .msi then Restart.
3: Run System File Checker Tool in Windows 10
If the issue persist, this may be due to insufficient power supplied to the portable SSD when the Surface goes to Standby/Sleep mode. If possible, try to use a Y-USB cable in connecting the SSD to the Surface. Using a Y-USB cable, allows you to plug the
extra end of the cable to the USB port on the power supply brick of the Surface which will give the SSD a 5watts power supplied.
Hope this helps.