Hi Jonathan,
About how to open the document(most Office documents like docx, xlsx, or pptx) in SharePoint Online, there is a related setting to control open within the browser or open in client applications. The setting is available in the libraries and sites. library
setting can inherit from the site or don't. When opening the documents in the search results, the opening method should follow the library setting where the files are.
About the behavior that open items in a new tab or new window, there should be a setting in the browser. Sometimes Outlook online will open emails in new windows, such as when you double-click the email.
About open in a new tab or in the same tab, that should depend on the content. Usually, if it's a web page like aspx file, or ducument, they can open in a new tab. Some items are not supported by the browser, like MSG file you have mentioned, it will follow
the protocol, download and open the related application.
About the 403 error you have mentioned, we might need more information to troubleshoot it. May I ask about what is the item that opens in the same tab so we could test from our side and analyze it?
Regards,
Alex