Hello,
I didn't use it for SharePoint but Application proxy and form based authentication can be use together (I used it to publish OWA for an Exchange onpremise)
The license requirement will be for application proxy which is Azure AD P1 minimum and based on this article for every user accessing the application via the application proxy.
For guest user the license requirement is a little bit different because it is based on a MAU (Monthly Active Users) which means your first 50000 will be free :
https://learn.microsoft.com/en-us/azure/active-directory/external-identities/external-identities-pricing
https://azure.microsoft.com/en-us/pricing/details/active-directory/external-identities/
Regards,