I found a solution as follows. Instead of linking the .xyz extension to a ProgID for the preview handler (XYZPreview.xyzfile), I link directly to the CLSID from the extension and removed XYZPreview.xyzfile entirely. Perhaps having two ProgIDs for the extension was causing the conflict (XYZFile.xyz & XYZPreview.xyzfile). The preview handler entries were modified as follows and both the file association and preview handler now work.
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\.xyz (Default)=""
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\.xyz\shellex\{8895B1C6-B41F-4C1C-A562-0D564250836F} (Default)="{64644512-C345-469F-B5FB-EB351E20129D}"
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D} (Default)="XYZ File Preview Handler"
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D} AppId="{534A1E02-D58F-44f0-B58B-36CBED287C7C}"
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D} DisableLowILProcessIsolation=1
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D}\InprocServer32\ (Default)="C:\Program Files (x86)\My App\XYZPreview.dll"
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D}\InprocServer32\ ProgID="XYZPreview.xyzfile"
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D}\InprocServer32\ ThreadingModel="Apartment"
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D}\InprocServer32\ VersionIndependentProgID="XYZPreview.xyzfile"
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{64644512-C345-469F-B5FB-EB351E20129D}\ProgID (Default)="XYZPreview.xyzfile"
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\PreviewHandlers {64644512-C345-469F-B5FB-EB351E20129D}="XYZ File Preview Handler"