Importing .ics file is not working

I am trying to import a calendar from my application to outlook calendar and for that, I made an end-point in my web app which is returning an iCal object(I tested it in postman and it is working fine and I am able to get the iCal object). But when I use this end-point I a not able to import my calendar into outlook
For outlook, it shows "Couldn't import calendar, try again later"
And after try again it shows the same thing, I don't know why it is happening
Few Points may be helpful for you:-
.) when I use ngrok.io to generate a link that points to my localhost server and using that link to import calendar then it works completely fine.
.) the domain which I am using is only accessible when I connect to it through VPN.