Hello @Gustavo Puente ,
You could browse to the Virtual Network Gateway, then Point-to-site configuration, and check the Allocated IP Addresses section at the bottom of that page to find out how many clients are connected or you could run the "Get-AzVirtualNetworkGatewayVpnClientConnectionHealth" PowerShell cmdlet as advised here.
Since the issue is happening intermittently and on few clients, it may require deeper investigation. Hence, if you have a support plan, I request you to file a support ticket, else please do let us know, we will try and help you get a one-time free technical support. In this case, could you send an email to azcommunity@microsoft.com referencing this thread as well as your subscription ID. Please mention "ATTN gishar" in the subject field.
Thank you for your cooperation on this matter and I look forward to your reply.