While it is true that Alma Linux is free and Open Source to download there is not currently an image that is directly provided by Azure. The only images available are published on the Marketplace by third party companies such as ProComputers or Cognosys. They are able to set the price per hour for the image which is why you are seeing a larger cost compared to RHEL which is a partner with Microsoft.
Since it is free and open source one option would be to launch and prepare a VM then upload it to Azure. Since this VM is uploaded by you want be charged an additional hourly rate. There is some preperation involved in this, you can review this page for the details.
Let me know if that makes sense or you have any more question.
Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.