Never noticed an Azure VM server is going in "hibernate/sleep mode".
Maybe you could check if the VPN Site to Site connection is "up and connected" all the time.
This could also be a reason for the first "ping timeouts", if the VPN Site to Site connection needs to be "started" first after a disconnect.
(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)
Regards
Andreas Baumgarten