I found the answer: set the web root to be a subdirectory, and then deploy everything to the original web root. This way, you can use the azure GitHub action to deploy.
However, I'm using PHP 8, as Drupal 9 requires it, and so the docs, which currently only describe Apache, don't apply because azure switched to nginx from PHP 8. Details on how to rewrite here.