I didn't use it for SharePoint but Application proxy and form based authentication can be use together (I used it to publish OWA for an Exchange onpremise)
The license requirement will be for application proxy which is Azure AD P1 minimum and based on this article for every user accessing the application via the application proxy.
For guest user the license requirement is a little bit different because it is based on a MAU (Monthly Active Users) which means your first 50000 will be free :
Your FBA user will exist in Active Directory and also in Azure via synchronization right ?
you may consult MS about license related question how it is suggested here: SharePoint 2019 on-prem licensing.