I'm assuming that Yanli is testing on SharePoint Online (please confirm @Yanli Jiang - MSFT ) and NOT SE combined with Office 2016.
On my system , SharePoint SE and Office 2016 , if I try to view/open an earlier version of a document from the version history dialog , I get the same "sorry couldn't open....VTI_History/<document name>" that I saw with SharePoint 2013 and Office 2016. This feature stopped working in the year 2019, up until this point I could open earlier versions and even compare different versions of the same document within Office.
My understanding (at that time) was that Windows itself was the problem as it is unable to open those _VTI__history URLs. I hoped that in SE that the version history wasn't using those VTI_History URLs anymore.
It appears that the same thing is happening even after having upgraded to SE!
Same issue when trying to open versions from the history in Office.
I have a recollection that earlier versions of documents stored in SharePoint Online WILL open in Office.
the workaround (in SharePoint on premises , e.g. SE) for anyone looking to view what a document looked like in an earlier version without having to restore is to right click the link for the earlier version and select the option "save link as" , this enables you to download a copy of the document at specific version to your local machine.
Would appreciate any guidance from MSFT as to how we can configure the experience that we used to have when opening earlier versions (i.e. so we don't see this "sorry could not open.." message )