DPI_AWARENESS_SYSTEM_AWARE, the application will be automatically scaled up or down by the system whenever the DPI changes from the system value. Maybe the layered window's off-screen buffer causes the curious behavior. Try to use
UpdateLayeredWindow instead of SetLayeredWindowAttributes and SetWindowPos.
It is recommended that desktop applications be updated to use per-monitor DPI awareness mode. When an application reports to Windows that it wants to run in this mode, Windows will not bitmap stretch the application when the DPI changes, instead sending WM_DPICHANGED to the application window. It is then the complete responsibility of the application to handle resizing itself for the new DPI. more information and a sample.
It is recommended that you set the process-default DPI awareness via application manifest. See Setting the default DPI awareness for a process for more information.