Sorry for the misleading question. I was fooled by strange errors shown during installation. It looks like I'm required to install .NET 4.7 on target system too, even if I'm only using .NET Core 5. Without .NET 4.7 (and ASP.NET) installer fails.
Now I only need to find out why there are problems in generated config files.
For example web.config contains something like that:
Unfortunatelly executable is placed in the same folder as web.config so file can't obviously be found.
Strictly speaking this is not related to creating setup, as the same folder is visible in the folder where my applicaiton is published before creating setup.