If you hacked away at the .msi to modify the installer, then you're pretty much on your own to troubleshoot problems. Both the installation process and the functionality of the app itself. The installer is telling you that "we don't support this OS".
Why don't you contact whoever supports/develops Portal v1.6.77 and get an updated version that they have documented to support IIS 10 on Server 2019.
Does anything show up in add/remove programs? Can you uninstall it?
As far as "where is this coming from?" you could start with some tools from https://Sysinternals.com.
The autoruns utility will report on all settings within Windows where a process could get launched at boot time. Services, scheduled tasks, registry keys, etc. You will need to run that and look for the setup or msi name in one of those entries.
If you can't find anything, then try Process Explorer. The process list will be indented by parent/child process. You can click and hold on the target icon (next to the binoculars icon) on the menu and drag and drop it on the installer windows (if there is one). Look to see the command line that launches the install process and also its parent processes. Do you see anything that might indicate how it was launched?