There may be files on that VHDX that are located closer to the end of the volume and are currently in use, which prevents defrag to optimize them and makes it impossible for you to shrink the volume. I would highly recommend you restart the virtual machine and try once again to see if that makes any difference. Alternatively, just shut down the virtual machine, attach the VHDX file in question to a different VM or the host directly, and perform the shrink. That should work.
Depending on what kind of data is hosted inside this virtual machine, you may also consider using Windows Deduplication to reduce storage consumption. It works exceptionally well on file servers. You can estimate your storage savings using a free analyzer tool https://www.starwindsoftware.com/starwind-deduplication-analyzer and see if it brings you even more storage gains than the shrink.