Finally found a solution:
https://github.com/WhitewaterFoundry/Pengwin/issues/498
https://developercommunity.visualstudio.com/content/problem/417332/uwp-creating-app-packages-for-store-package-does-n.html
Basically, when VS 2019 builds a package, it fails to include some visual assets if the solution is in OneDrive synced folder. I copied the solution folder to desktop (folder not included in OneDrive) and rebuild the solution and it worked.
A couple notes so that if anyone else has the same issue, they will find this thread when searching.
I got warnings like the ones in the post linked post:
GENERATEPROJECTPRIFILE : warning PRI298: 0xdef00535 - File or folder ignored due to file attributes (hidden, system, link): ARM64\Release\PackageUploadLayout\Assets\Square44x44Logo.scale-400.png.
I checked to see if any file or folder had hidden or system attribute, but that was not the case. Plus, there were plenty of images inside the overall package that I did not think these warnings meant anything (had warnings for images that were included in the package).
Hope this helps others as well. One of the posts states this issue started with Visual Studio 2019 and was fine in Visual Studio 2017.
Thanks FayWang to trying to help!