Hi Rupesh,
You can find the separate window licensing cost in Azure pricing calculator, there from you can segregate the compute cost like Compute Cost = Total Cost - Licensing cost.
For example, Below, I choose the SQL Server Windows VM in pricing calculator and there is pricing segregation Microsoft provide.
https://azure.microsoft.com/en-in/pricing/calculator
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.