is it mandatory to use Entity Framework in a Core MVC project?
Use ADO.NET if you like. Keep in mind, EF Core is built on top of ADO.NET. EF Core can easily call a stored procedure.
can I simply use database connections to retrieve and update data using existing stored procedures?
Yes. ASP.NET Core has dependency injection to pass a configuration service to a constructor. The configuration service is used to get to the connection string from appsettings.json.
ASP.NET Core fundamentals overview
Dependency injection in ASP.NET Core
I just started reading about MVC and it seems that there are lots of differences between types of projects and versions of .NET: some projects provide a Data folder by default with an ApplicationDbContext.cs class, some are not providing the Data folder. Some older versions use to have a Startup.cs, currently everything moved to Program.cs.
Correct ASP.NET Core is very different from ASP.NET framework. Set aside some time to read the ASP.NET Core fundamentals overview and go through a few tutorials to get a feel for ASP.NET Core.
Razor Pages with Entity Framework Core in ASP.NET Core - Tutorial 1 of 8