I was missing app.Endpoints as my middleware, updated my program.cs as below:
app.UseAuthentication(); // For identity
app.UseRouting();
app.UseAuthorization(); // For identity
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
endpoints.MapRazorPages();
});