Azure function with python app, seeing: Error 404: When request Azure resource at PublishContent, zipDeploy : Failed to use /home/runner/work/_temp/temp_web_package_...zip as ZipDeploy content

Question

We are running several Azure functions with python and randomly see this 404 error when deploying via github. These github workflows have worked 2 years for us to deploy various python functions hundreds of times to azure and now are suddenly no longer reliable

   - name: Publish Azure Function
        uses: Azure/functions-action@v1
        id: fa
        with:
          app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
          package: ${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}
          publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }}

While debugging now we also tried v1.5.1, but no success with it either. Some more or less random restart/delete/disable may or may not help to resolve it (as described here microsoft/azure-pipelines-tasks#18739 (comment)) but the issue comes back and then one can start the deployment over and over again and see the same error. Here is how the github job log starts with debug logging:

Current runner version: '2.311.0'
Operating System
Runner Image
  Image: ubuntu-22.04
  Version: 20240116.3.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240116.3/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240116.3
Runner Image Provisioner
  2.0.324.1

And this is how it fails:

##[debug]Deployment status: 0 'Building and Deploying '***-4e62-871a-d8bdc4ef87f0'.'. retry after 5 seconds
##[debug]setting affinity cookie ["ARRAffinity=****a443cf3f6c7c2e4daaee92c2062af1b69e3649bf26d43fe66fe700a7de;Path=/;HttpOnly;Secure;Domain=***.scm.azurewebsites.net","ARRAffinitySameSite=****443cf3f6c7c2e4daaee92c2062af1b69e3649bf26d43fe66fe700a7de;Path=/;HttpOnly;SameSite=None;Secure;Domain=***.scm.azurewebsites.net"]
##[debug][GET] [https://***.scm.azurewebsites.net:443/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY_FUNCTIONS_V1&time=2024-01-23_11-20-47Z](https://***.scm.azurewebsites.net/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY_FUNCTIONS_V1&time=2024-01-23_11-20-47Z)
##[debug]Could not parse response body.
##[debug]{}
Error: Failed to deploy web package to App Service.
Error: Execution Exception (state: PublishContent) (step: Invocation)
Error:   When request Azure resource at PublishContent, zipDeploy : Failed to use /home/runner/work/_temp/temp_web_package_8472975363497814.zip as ZipDeploy content
Error:     Failed to deploy web package to App Service.
Not Found (CODE: 404)
Error:       Error: Failed to deploy web package to App Service.
Not Found (CODE: 404)
    at Kudu.<anonymous> (/home/runner/work/_actions/Azure/functions-action/v1/lib/appservice-rest/Kudu/azure-app-kudu-service.js:235:41)
    at Generator.throw (<anonymous>)
    at rejected (/home/runner/work/_actions/Azure/functions-action/v1/lib/appservice-rest/Kudu/azure-app-kudu-service.js:6:65)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Error: Deployment Failed!
##[debug]Node Action run completed with exit code 1

It does seem to finish the publishing though (as our version tag shows the new version) but it does not start the remaining steps of the deployment as it considers this step as failed.