Hello @Anonymous , thank you for reaching out. You would just need either an Azure AD P1 or an Azure AD P2 license for the administrator, for him to configure the Azure AD App Proxy configurations but you need Azure Premium license for any user that is using app proxy
You can find more details on the same here: https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/application-proxy-faq#why-is-the-enable-application-proxy-button-grayed-out
Hope this helps.
Do let us know if this helps and if there are any more queries around this, please do let us know so that we can help you further. Also, please do not forget to accept the response as an Answer; if the above response helped in answering your query.