Hi,
Yes, you can install SP3 on Exchange server 2010 RTM.
For coexistence with Exchange 2016, Update Rollup 11 for Exchange 2010 SP3 or later to be installed.
Also, for Exchange 2016, all the domain controllers in the forest need to be running Windows Server 2008 R2 or above and the forest functional level is Windows Server 2008 R2 or higher.
Please note that Exchange 2010 support has already been ended.
https://learn.microsoft.com/en-us/exchange/plan-and-deploy/system-requirements?view=exchserver-2016
https://learn.microsoft.com/en-us/lifecycle/announcements/exchange-server-2010-support-extended#:~:text=Please%20go%20here%20to%20search,2010%20and%20SharePoint%20Server%202010.
If the above suggestion helps, please click on "Accept Answer" and upvote it.