1. First Step is to install pre-requisites and enable required roles and features for Office online Server:

 Add-WindowsFeature Web-Server,Web-Mgmt-Tools,Web-Mgmt-Console,Web-WebServer,Web-Common-Http,Web-Default-Doc,Web-Static-Content,Web-Performance,Web-Stat-Compression,Web-Dyn-Compression,Web-Security,Web-Filtering,Web-Windows-Auth,Web-App-Dev,Web-Net-Ext45,Web-Asp-Net45,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Includes,InkandHandwritingServices,NET-Framework-Features,NET-Framework-Core,NET-HTTP-Activation,NET-Non-HTTP-Activ,NET-WCF-HTTP-Activation45,Windows-Identity-Foundation,Server-Media-Foundation
 

 Add-WindowsFeature Web-Server,Web-Mgmt-Tools,Web-Mgmt-Console,Web-WebServer,Web-Common-Http,Web-Default-Doc,Web-Static-Content,Web-Performance,Web-Stat-Compression,Web-Dyn-Compression,Web-Security,Web-Filtering,Web-Windows-Auth,Web-App-Dev,Web-Net-Ext45,Web-Asp-Net45,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Includes,NET-Framework-Features,NET-Framework-45-Features,NET-Framework-Core,NET-Framework-45-Core,NET-HTTP-Activation,NET-Non-HTTP-Activ,NET-WCF-HTTP-Activation45,Windows-Identity-Foundation,Server-Media-Foundation
 

• NET Framework 4.5.2 (If .Net Framework 4.5.2 or later is already installed, you can ignore this)
• Visual C++ Redistributable Packages for Visual Studio 2013
• Visual C++ Redistributable for Visual Studio 2015
• Microsoft.IdentityModel.Extention.dll

         New-OfficeWebAppsFarm -InternalURL "https://servername" -AllowHttp -EditingEnabled

1. To verify that the Farm is created, we can try opening the Wopi Discovery XML file by accessing URL: https://servername/hosting/discovery

       Note: Configuring OOS Farm using HTTP is not recommended for Production Farms.

   b. For configuring a single-server farm that use HTTPS, run the following command

2. New-OfficeWebAppsFarm -InternalUrl "https://server.yourdomain.com" -ExternalUrl "https://hostname.yourdomain.com" -CertificateName "OfficeWebApps Certificate" -EditingEnabled
    

   Note that -External URL is the URL that should be accessible on internet and -Internal URL is the FQDN of the OOS Server.

   Do not use Self Signed Certificate here. Use a Certificate obtained from a valid Certificate Authority.

   To verify that the OOS Farm is created, access the URL: https://server.yourdomain.com/hosting/discovery. The WOPI Discovery XML File should be accessible.

         c. For configuring a multi-server load balanced OOS Farm using HTTPS:
          On the First Server in the Farm run the following command:

         New-OfficeWebAppsFarm -InternalUrl "https://server.yourdomain.com" -ExternalUrl "https://hostname.yourdomain.com" -SSLOffloaded -EditingEnabled
    

         This is in assumption that you have SSL Offloading in place on NLB and certificates are placed on NLB.        On the remaining servers which you want to join to this OOS farm, run the following command:

        New-OfficeWebAppsMachine -MachineToJoin "server1.yourdomain.com"
    

   Note: We need to include -EditingEnabled in the New-OfficeWebAppsFarm command line to enable users to edit the documents. Once the Servers are joined to the farm, we are good to integrate the SharePoint Farm and OOS Farm.
3. 4. Integrating OOS with your SharePoint Farm:
4. Once you have the OOS Farm up, you can proceed integrating SharePoint Farm with OOS. Please note that you need to have claims aware web applications for the Office Online rendering and editing to work. Also make a note that System Account won't be able to preview or open the document via OOS. Make sure you use an account other that the System Account.
5.     a. If you have a single-server OOS Farm that use HTTP:          Open SharePoint Management shell using Administrator and run the following command:

            New-SPWOPIBinding -ServerName <OOSServerName> -AllowHTTP
    

        Since you are using HTTP to communicate to the host in OOS(see step 3.a), the WopiZone should be specified as HTTP.      To verify which zone you are using in SharePoint, use command Get-SPWopiZone. If the output returned for this command is internal-https, change it to internal-http by running the following command:

           Set-SPWOPIZone -zone "internal-http"
    

     Also, since you are using HTTP, we need to allow OAuth over HTTP in the security token config as well.

             $config = (Get-SPSecurityTokenServiceConfig)
 

                $config.AllowOAuthOverHttp = $true
 

                $config.Update()
 

              New-SPWOPIBinding -ServerName <OOSServerhostname>

                 Set-SPWOPIZone -zone "external-https"
 

                 Remove-SPWOPIBinding -All:$true
 

While applying changes to the Search Topology, the following error is generated "Errors were encountered during the configuration of the Search service application" and the following traces will be populated in ULS "System.OverflowException: Arithmetic operation resulted in an overflow"

ULS Stack:

SearchConfigWizardFinish page failed with exception. System.OverflowException: Arithmetic operation resulted in an overflow.    Server stack trace:    
  at Microsoft.Office.Server.Search.Administration.UnsafeNativeMethods.NetLocalGroupGetMembers(String groupName)   
  at Microsoft.Office.Server.Search.Administration.UnsafeNativeMethods.TraceLocalGroups()   
  at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Provision()   
  at Microsoft.Office.Server.Search.Administration.SearchAdminUtils.DeployLocalServiceInstance[T](T localSearchServiceInstance)   
  at Microsoft.Office.Server.Search.Administration.SearchConfigurationJobDefinition.Create(AbstractSearchConfigWizard wizardOM)   
  at Microsoft.Office.Server.Search.Internal.UI.SearchConfigWizardFinish.ExecuteConfigurationTimerJob(SPLongOperationState longOperationState, String& strRedirectPage)   
  at Microsoft.SharePoint.SPStatefulLongOperation.SetThreadCultureAndRun(RunStatefulOperation runOperation, SPLongOperationState state, CultureInfo culture, CultureInfo uICulture)   
  at Microsoft.SharePoint.SPStatefulLongOperation.<>c__DisplayClass6.<Run>b__0(Object state)   
  at Microsoft.SharePoint.Utilities.SPThreadPool.WaitCallbackWrapper(Object state)   
  at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)   
  at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)    Exception rethrown
 at [0]:    
  at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)   
  at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)   
  at System.Threading.WaitCallback.EndInvoke(IAsyncResult result)   
  at Microsoft.SharePoint.Utilities.SPThreadPool.RunAsyncWorkItem(ImpersonatedCallbackParam param, TimerCallback timerCallback, Object timerState, Int32 timerPeriod)   
  at Microsoft.SharePoint.Utilities.SPThreadPool.RunAsyncWorkItemWithImpersonation(WaitCallback workItemCallback, Object workItemState, TimerCallback timerCallback, Object timerState, Int32 timerPeriod)   
  at Microsoft.SharePoint.SPStatefulLongOperation.Run(RunStatefulOperation runOperation)   
  at Microsoft.Office.Server.Search.Internal.UI.SearchConfigWizardFinish.<>c__DisplayClass2.<Commit>b__0(SPStatefulLongOperation longOperation) 

Cause

The error was thrown at Microsoft.Office.Server.Search.Administration.UnsafeNativeMethods.NetLocalGroupGetMembers(String groupName). From this we could figure out that it errors out while enumerating members of one of the local group. On a closer review of the ULS logs I was able to figure out the problematic group:

w3wp.exe (0x0C3C) 0x16CC SharePoint Server Search Administration dmg1 High Group WSS_WPG

 w3wp.exe (0x0C3C) 0x16CC SharePoint Server Search Administration dmb2 High The call to SearchServiceInstance.Provision (server 'servername') failed. Setting back to previous status 'Disabled'. System.OverflowException: Arithmetic operation resulted in an overflow.     at Microsoft.Office.Server.Search.Administration.UnsafeNativeMethods.NetLocalGroupGetMembers(String groupName)     at Microsoft.Office.Server.Search.Administration.UnsafeNativeMethods.TraceLocalGroups()     at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Provision()     at Microsoft.Office.Server.Search.Administration.SearchAdminUtils.DeployLocalServiceInstance[T](T localSearchServiceInstance)

The issue was caused due to the substantial number of users added to the WSS_WPG group.

Resolution

Removed the individual users from WSS_WPG group. If you want to add the users to WSS_WPG group, create an AD Security Group, add the users to this SG and add the SG to the WSS_WPG group.

Applies to: SharePoint Server 2013, SharePoint Server 2016