Dynamically Maintain the IoT Edge Module Version in Azure DevOps Pipeline using Layered Deployment

Dhanavath Vishnu 366 Reputation points
2023-05-29T10:34:26.5833333+00:00

Hi Team,

Hello @LeelaRajeshSayana-MSFT @satishbodduakon

Greetings for the day!!!!!

I have a scenario in my project, where we are having 5 IoT Edge modules (3 Backend, 1 Frontend, 1 Microsoft developed).

We are creating the manual deployment in the azure portal using the module image versions. Once the local changes are done by developers in visual studio, before committing the code to the branch we use to change the module version in the module.json file of the specific module and commit the changes. Using the deployment template depending on the platform (amd64, arm), we build and push the modules from visual studio. Once the images are pushed to ACR, using the image tags, we create a deployment template with the tags and target the Dev/QA environment Edge devices. This way we were able to achieve the latest images running on the specified environments.

While automating this entire process through azure devops pipeline, we have used the Azure IoT Edge tasks. We have created individual tasks for the modules, the tasks internally will refer to that version of the module image as defined in the module.json file and build that version of the image and push it to the ACR. The deployment we prefer to use in our project is Layered deployment. Using the Azure CLI tasks in the pipeline, we can create Layered deployment and maintain its creation and updating in the azure portal dynamically.

The Problem we are facing is,

how to read the dynamic change of the image version in module.json file using the pipeline tasks, as we won’t be maintaining the previous pipeline run variable values,

for Example: In previous build we have used version of 1.0.0, now in the next commit by developer, it changes to 1.0.1. As we won’t be having any previous versions reference in my pipeline, we can’t catch the latest (1.0.1) version of the image in the layered deployment, as this image updating automatically happens in the standard deployment template using Azure DevOps Pipeline.

Extension to this one more requirement is, if I want to build the specific module image (only module 1), tasks related to the other modules to be skipped in the pipeline, we are planning to use Condition: into the tasks those related to the modules build, push, replace image versions. For applying these conditions to the tasks, we are planning to maintain the Boolean variables depending upon the module image version change. If the image change is deducted, make the variable as ‘true’, using this variable the defined condition(eq.variables, true) become true for only to those tasks which are specific to that module images.

Problems:

  1. How to read the change in the image version from module.json file, to replace these image versions in the layered deployment template using Azure CLI tasks.
  2. Once the image versions are read, If the change in the version found, the defined Boolean variables in the pipelines should be set to the true condition, which internally trigger the related tasks only, rather than executing all tasks in the pipeline.

Ex: Here I have defined my sample yaml file for maintaing the multiple variables to catch the image change, update the Boolen variable accordingly and using condition to run the specific module tasks


#Image Version Change variable, need to update this variable in the pipeline, when there is image version change is deducted. 
Module1_IMAGE_VERSION: 1.0.0-test 
Module2_IMAGE_VERSION: 1.0.0-test 
Module3_IMAGE_VERSION: 1.0.0-test 
#Image Version Change Boolean variable to execute the only related tasks, rather than all tasks to execute in the pipeline.     
Module1_IMAGE_Change: false
Module2_IMAGE_Change: false
Module3_IMAGE_Change: false

build Module1 Image
- task: AzureIoTEdge@2
  inputs:
    action: 'Build module images'
    templateFilePath: '$(ROOT_DIR)/Module1/deployment.template.json'
    defaultPlatform: $(BUILD_PLATFORM))
  displayName: Build Edge Module Module3 Image
  condition: eq(variables.Module1_IMAGE_Change, true
# build Module1 Image
- task: AzureIoTEdge@2
  inputs:
    action: 'Build module images'
    templateFilePath: '$(ROOT_DIR)/Module1/deployment.template.json'
    defaultPlatform: $(BUILD_PLATFORM)
  displayName: Build Edge Module Module3 Image
  condition: eq(variables.Module1_IMAGE_Change, true)

# build Module2 Image
- task: AzureIoTEdge@2
  inputs:
    action: 'Build module images'
    templateFilePath: '$(ROOT_DIR)/Module2/deployment.template.json'
    defaultPlatform: $(BUILD_PLATFORM)
  displayName: Build Edge Module Module2 Image
  condition: eq(variables.Module2_IMAGE_Change, true)

# build Module3 Image
- task: AzureIoTEdge@2
  inputs:
    action: 'Build module images'
    templateFilePath: '$(ROOT_DIR)/Module3/deployment.template.json'
    defaultPlatform: $(BUILD_PLATFORM)
  displayName: Build Edge Module Module3 Image
  condition: eq(variables.Module3_IMAGE_Change, true)



Let us know your inputs/suggestions on the same, is the method we are using is right method or should we need to change anywhere, please guide us, if you have any reference docs, please do share with us.

Thanks

D. Vishnu

Azure IoT
Azure IoT
A category of Azure services for internet of things devices.
373 questions
Azure IoT Edge
Azure IoT Edge
An Azure service that is used to deploy cloud workloads to run on internet of things (IoT) edge devices via standard containers.
525 questions
Azure IoT Hub
Azure IoT Hub
An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.
1,093 questions
{count} votes

Accepted answer
  1. QuantumCache 20,021 Reputation points
    2023-05-30T16:30:23.79+00:00

    Hello @Dhanavath Vishnu

    Please refer to the below YML script!

    Frist Time RUN: the variable is updated and the boolean flag output is 'True'

    User's image

    Second time run: Few stages are skipped due to variable read as 'False'

    User's image

    Please update the values of Group name, PAT token and Org URL.

    trigger:
    - master
    pool:
      vmImage: ubuntu-latest
    
    variables:
    - group: AzureIoTEdge-Dev # Link the variable group  
    
    stages:
    - stage: Stage1
      jobs:
      - job: Stage1_Job1
        steps:
        - task: PowerShell@2
          displayName: 'Stage1-Job1-Task1'
          name: 'Stage1_Job1_Task1'
          inputs:
            targetType: 'inline'
            script: |
              $path="module.json"
              $content = Get-Content $path
              $version = ($content | ConvertFrom-Json).image.tag.version
              Write-Host "##vso[task.setvariable variable=Module1_IMAGE_VERSION;isOutput=true]$version"
              Write-Host "Stage1_Job1_Task1: Current Version read from the Repo file is : $version"
              Write-Host "##vso[task.setvariable variable=currentVersion;isOutput=true;]$version"
    
      - job: Stage1_Job2
        dependsOn: Stage1_Job1
        variables:
          NewVersion: $[dependencies.Stage1_Job1.outputs['Stage1_Job1_Task1.currentVersion']]
        steps:
        - task: PowerShell@2
          displayName: 'Stage1-Job2-Task1'
          inputs:
            targetType: 'inline'
            script: |
              Write-Host "Stage1_Job2: The value of currentVersion read from Stage1_Job1_Task1 is $(NewVersion)"
              Write-Host "Stage1_Job2: The value of Predefined variable PreviousVersion read from Pipeline Variables is: $(PreviousVersion)"
        
        - task: PowerShell@2
          displayName: 'Stage1-Job2-Task2'
          name: 'Stage1_Job2_Task2'
          inputs:
            targetType: 'inline'
            script: |
              Write-Host "Stage1_Job2_Task2: Checking if image version has changed..."
              if ("$(NewVersion)" -ne "$(PreviousVersion)") {
              $connectionToken="qh6q"
              $base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
              $URL = "https://dev.azure.com/{}/AzureIoTEdge/_apis/distributedtask/variablegroups/{}?api-version=6.0-preview.1"
              
              $body = '{"id":2,"type":"Vsts","name":"AzureIoTEdge-Dev","variables":{"PreviousVersion":{"isSecret":false,"value":"$(NewVersion)"}}}'
              
              $Result = Invoke-RestMethod -Uri $URL -Headers @{authorization = "Basic $base64AuthInfo"} -Method Put -Body $body -ContentType "application/json"
              $Variable = $Result.value.variables | ConvertTo-Json -Depth 100
              Write-Output $Variable
              Write-Host "##vso[task.setvariable variable=MODULE1_IMAGE_CHANGE;isOutput=true]true"
              Write-Host "Stage1_Job2_Task2: Checking if image version has changed...$MODULE1_IMAGE_CHANGE " 
               } else {
                Write-Host "##vso[task.setvariable variable=MODULE1_IMAGE_CHANGE;isOutput=true]false"
              }
    
      - job: Stage1_Job3
        dependsOn: Stage1_Job2
        variables:
          Module1ImageChange: $[dependencies.Stage1_Job2.outputs['Stage1_Job2_Task2.MODULE1_IMAGE_CHANGE']]
        steps:
        - task: PowerShell@2
          displayName: 'Stage1-Job3-Task1'
          condition: eq(variables['Module1ImageChange'], 'true')
          inputs:
            targetType: 'inline'
            script: |
              Write-Host "Stage1_Job3_Task1: The value of Module1ImageChange is true, running new tasks..."
    
        - task: PowerShell@2
          displayName: 'Stage1-Job3-Task2'
          condition: ne(variables['Module1ImageChange'], 'true')
          inputs:
            targetType: 'inline'
            script: |
              Write-Host "Stage1_Job3_Task2: The value of Module1ImageChange is false, skipping new tasks..."
    
      - job: Stage1_Job4
        dependsOn: Stage1_Job2
        variables:
          Module1ImageChange: $[dependencies.Stage1_Job2.outputs['Stage1_Job2_Task2.MODULE1_IMAGE_CHANGE']]
        steps:
        - task: PowerShell@2
          displayName: 'Stage1-Job4-Task1'
          condition: eq(variables['Module1ImageChange'], 'true')
          inputs:
            targetType: 'inline'
            script: |
              Write-Host "Stage1_Job4_Task1: The value of Module1ImageChange is true, running new tasks..."
    
        - task: PowerShell@2
          displayName: 'Stage1-Job4-Task2'
          condition: ne(variables['Module1ImageChange'], 'true')
          inputs:
            targetType: 'inline'
            script: |
              Write-Host "Stage1_Job4_Task2: The value of Module1ImageChange is false, skipping new tasks..."
    
    

    If the response is helpful, please click "Accept Answer" and upvote it. So that we can close this thread.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.


    Comments have been turned off. Learn more