You'll register your MVC application with the OAuth/OIDC service you select for your project. This involves creating a secret that only the OAuth/OIDC service and your MVC project knows as well as other bit of information like a your web application redirect URL.
1) Does ASP.NET Core MVC UI application needs cookie authentication even if want it to use it as an an UI which will be calling web api(implemented with OAuth)
Cookie authentication authorizes the browser when accessing MVC actions. Web API client submits a bearer token which authorizes access to Web API actions. Keep in mind, there can be many types of clients and several OAuth/OIDC flows. Typically you'll pick a flow that fits your security needs. Do a google search for OAuth/OIDC.
2)Can I implement oAuth mechanism for ASP.NET Core MVC Web App UI & WEB API on same server & same project. If yes, how to achieve that ?
Of course. Pick an OAuth/OIDC provider. Read the OAuth/OIDC provider documentation and implement. Well known OAuth/OIDC providers will have a library you can use in your project.
3)Can I implement oAuth mechanism for ASP.NET Core MVC Web App UI & WEB API on different server & different project. If yes, how to achieve that ?
Yes. Same as 2. One server or multiple servers has no effect. However, it is common to use separate servers due to security.
4) Does ASP.NET MVC always needs cookie in the picture. And when cookie comes, I guess session will be there on server.
Cookies and Session are two different middleware services. As explained in 1 above, cookie authentication authorizes the browser to access an MVC application.
The official documentation covers these concepts quite extensively.
ASP.NET Core security topics
Overview of ASP.NET Core authentication
Introduction to authorization in ASP.NET Core