Hello @Adam Steven ,
Thanks for reaching out.
This is still not supported by the office-addin-sso default package. You have to remove that default package and then add the code yourself to handle SSO so that you can deploy to Azure. Our product team have a plan to document this in August, so hopefully will have guidance around this soon.
Here are some sample JavaScript to deploy to Azure app service.
Reference:
https://github.com/OfficeDev/office-js-docs-pr/issues/2279#issuecomment-733153415
https://github.com/OfficeDev/office-js-docs-pr/issues/2344#issuecomment-823642553
Hope this helps.
Regards,
Siva