Azure Function Apps oryx build continues to run even when ENABLE_ORYX_BUILD=false

Question

We currently deploy to our azure function instances from CircleCI. As part of our pipeline we build the applications and output them as a package. Our Azure network is behind a VPN, therefore we use the method of deploying our zip to a blob container, then sending the following api request in order to trigger the deployment:

            SITE_URI="https://management.azure.com/subscriptions/${AZURE_SUBSCRIPTION}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Web/sites/${APP_NAME}/extensions/onedeploy?api-version=2020-12-01"
            az rest --method PUT \
              --uri $SITE_URI \
              --body '{ 
                  "properties": { 
                      "properties": {
                          "packageUri": "'"${ZIP_URL}"'"
                      }, 
                      "type": "zip", 
                      "ignorestack": false,
                      "clean": true,
                      "restart": true
                  }
              }'

This all worked perfectly for several months until roughly two weeks ago, when all of our builds suddenly stopped deploying correctly. When debugging the issue we noticed that our deployment had randomly switched to using Oryx, which is wasnt previously using. Our last previously working deployment logs

2022-02-17T15:01:00  Fetching changes.
2022-02-17T15:01:00    Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/09c22093-cd1d-4945-a749-f46af7984b92.zip (2.25 MB) to /tmp/zipdeploy/extracted
2022-02-17T15:01:03  Updating submodules.
2022-02-17T15:01:03  Preparing deployment for commit id '14a12017a6'.
2022-02-17T15:01:04  Running build. Project type: OneDeploy
2022-02-17T15:01:04  Copying the manifest
2022-02-17T15:01:04  Clean deploying to /home/site/wwwroot
2022-02-17T15:01:04  Generating deployment script.
2022-02-17T15:01:04    Using cached version of deployment script (command: 'azure -y --no-dot-deployment -r "/tmp/zipdeploy/extracted" -o "/home/site/deployments/tools" --basic --sitePath "/tmp/zipdeploy/extracted"').
2022-02-17T15:01:04  Running deployment command...
2022-02-17T15:01:04    Command: "/home/site/deployments/tools/deploy.sh"
2022-02-17T15:01:04    Handling Basic Web Site deployment.
2022-02-17T15:01:05    Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2022-02-17T15:01:05    Copying file: 'host.json'
2022-02-17T15:01:05    Copying file: 'DispatchIdeasPushNotifications/function.json'
2022-02-17T15:01:05    Copying file: 'GetIdeasNotifications/function.json'
2022-02-17T15:01:05    Copying file: 'HttpTriggerJob/function.json'
2022-02-17T15:01:05    Copying file: 'ProcessAndSendPushNotification/function.json'
2022-02-17T15:01:05    Copying file: 'QueueSendPushNotification/function.json'
2022-02-17T15:01:05    Copying file: 'SendIdeaPushNotification/function.json'
2022-02-17T15:01:05    Copying file: 'SendPushNotification/function.json'
2022-02-17T15:01:05    Copying file: 'bin/Notifications.dll'
2022-02-17T15:01:05    Copying file: 'bin/Notifications.pdb'
2022-02-17T15:01:05    Copying file: 'bin/extensions.json'
2022-02-17T15:01:05    Copying file: 'bin/function.deps.json'
2022-02-17T15:01:05    Finished successfully.
2022-02-17T15:01:05  Build completed succesfully.
2022-02-17T15:01:05  Running post deployment command(s)...
2022-02-17T15:01:05  Requesting site restart
2022-02-17T15:01:05  Requesting site restart. Attempt #1
2022-02-17T15:01:06  Successfully requested a restart. Attempt #1
2022-02-17T15:01:06  Deployment successful.

Our current build logs:

    2022-02-23T13:07:29  Fetching changes.
    2022-02-23T13:07:29    Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/831928c1-bb64-4766-929d-9ba35305391d.zip (2.25 MB) to /tmp/zipdeploy/extracted
    2022-02-23T13:07:44  Updating submodules.
    2022-02-23T13:07:45  Preparing deployment for commit id 'c3d03453-6'.
    2022-02-23T13:07:45  Repository path is /tmp/zipdeploy/extracted
    2022-02-23T13:07:45  Running oryx build...
    2022-02-23T13:07:45    Command: oryx build /tmp/zipdeploy/extracted -o /tmp/build/expressbuild --platform dotnet --platform-version 2.2 -i /tmp/8d9f6cd7b91f675
    2022-02-23T13:07:47    Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
    2022-02-23T13:07:47    You can report issues at https://github.com/Microsoft/Oryx/issues
    2022-02-23T13:07:47    
    2022-02-23T13:07:47    Oryx Version: 0.2.20211207.1, Commit: 46633df49cc8fbe9718772a3c894df221273b2af, ReleaseTagName: 20211207.1
    2022-02-23T13:07:47    
    2022-02-23T13:07:47    Error: Platform 'dotnet' has been disabled.
    2022-02-23T13:07:47    Build Operation ID: |1bFAFzqgpVs=.d3f750a2_
    2022-02-23T13:07:47    Repository Commit : c3d03453-6c99-48c3-b922-c72ea8b6bb91
    2022-02-23T13:07:47    
    2022-02-23T13:07:47    Detecting platforms...
    2022-02-23T13:07:47    Platform 'dotnet' has been disabled, so skipping detection for it.
    2022-02-23T13:07:47    Could not detect any platform in the source directory.
    2022-02-23T13:07:49    Error: Platform 'dotnet' has been disabled.
    /opt/Kudu/Scripts/starter.sh oryx build /tmp/zipdeploy/extracted -o /tmp/build/expressbuild --platform dotnet --platform-version 2.2 -i /tmp/8d9f6cd7b91f675
    2022-02-23T13:07:49  
    2022-02-23T13:07:50  Deployment Failed.

For our regular App Services we have been able to revert to the previous build behaviour by disabling the Oryx build explicitly using ENABLE_ORYX_BUILD=false. This has unblocked most of our services, however all three of our Azure Function Apps continue to use Oryx even with ENABLE_ORYX_BUILD=false set in the application settings.

This issue has severely impacted our companies ability to deploy updates to our systems for multiple weeks now and we have been unable to resolve it. Any guidance or advice you could give to help unblock our teams would be greatly appreciated as we're kinda at a loss on what else we can attempt.

Answer 1

After some fiddling around I've finally found a workaround for the issue.

It's documented in the documentation of Oryx Configration as a way to completely bypass automatic build, so should be a pretty safe workaround to rely on in the future.

provide an alternate build-and-run script. Bypasses automatic build completely.

For it to work, in the Zip file you want to deploy, create a file called startup_script.sh with the following content:

#!/bin/bash
echo "Bypassing oryx"

Next, in your function app, set the COMMAND environment variable to /tmp/zipdeploy/extracted/startup_script.sh before your next deployment attempt.

For me, this forced Oryx to use this script, bypass the automatic build, and deploy the function app successfully.

Thanks to Martin for having documented this mechanism in the context of webapps here: https://www.blimped.nl/fixing-azure-webapp-zip-deployment-issue/

Answer 2

@Justin McManus ,

Thanks for posting the question in Q&A. I understand the criticality of this deployment issue.

Using the setting ENABLE_ORYX_BUILD=false should disable the Oryx from being used for build process. I would suggest you to add the following setting as well
SCM_DO_BUILD_DURING_DEPLOYMENT=false

SCM_DO_BUILD_DURING_DEPLOYMENT will bypass the automatic build process and use the Kudu service instead. Please try and let me know the outcome.

Answer 3

Hi @MughundhanRaveendran-MSFT ,

Thanks very much for your quick response.

We have disabled the SCM_DO_BUILD_DURING_DEPLOYMENT flag but we are still seeing Oryx runs for our Azure Function Apps.

Answer 4

Hi All, I kindly ask, was there a solution suggestion in addition to:

"Using the setting ENABLE_ORYX_BUILD=false should disable the Oryx from being used for build process. I would suggest you to add the following setting as well
SCM_DO_BUILD_DURING_DEPLOYMENT=false" ?

Answer 5

Hey @Christopher Richardson ,

I escalated the issue to the Senior Azure Support team through the help of @MughundhanRaveendran-MSFT . Unfortunately however we just went round in circles for 2 months before they finally told me that its a known issue and they are working on it. No timelines etc were given for resolution of the issue, so I think we just have to wait around and hope they fix it. In the meantime they advised manually pushing builds to the server from local machines, which is obviously not great, but thats all we can do :/