So after long research and help from MS support, Finally figured out the issue. As I said before, it is not related to how the container starts up as the container was starting up in less than 6 seconds. The issue we noticed is that when the start-up fails due to HTTP health-check timeout, the app is starting up with port 80 as the listening port. When it is successful, it starts up with port 8080.
Spring-Boot default listening port is 8080. The fix is to manually add the configuration for the APP service
App Setting Name: WEBSITES_PORT
Value: 8080
The above configuration seems to have fixed the issue and now the time to start is the time taken by the docker container within the app service.