@Erland Sommarskog Thanks for checking on this and giving me valuable feedback. I found the issue and managed to complete the in-place upgrade.
The link given on the error (https://go.microsoft.com/fwlink?LinkID=2209051&ProdName=Microsoft%20SQL%20Server&EvtSrc=setup.rll&EvtID=50000&ProdVer=16.0.1000.6&EvtType=0xDBB00D2A%25400x294A9FD9) is misleading as it lands on the incorrect page that is on ARC registration. The real problem was the feature "Machine Learning Services and Language Extensions". Since the upgrade wizard does not give you an option to untick this feature, I manually uninstalled the feature on the old version (using the install wizard for SQL server 2017) and then tried the in-place upgrade which went through without any issues. If I need, I think I can install the machine learning extension later on, which we don't need at this point.