Azure Development Community https://docs.microsoft.com/archive/blogs/azuredev/feed.xml We’re a community leadership team of Microsoft employees, MVPs, Partners, and Azure development experts. This blog represents the opinions and expertise of the broad development echosystem; it is not the officially endorsed position of Microsoft or Azure. Thu, 09 May 2019 17:33:15 GMT en-US hourly 1 Azure Development Community - Same Great Blog... NEW HOME!!! https://docs.microsoft.com/archive/blogs/azuredev/azure-development-community-same-great-blog-new-home Thu, 28 Feb 2019 15:13:05 GMT https://blogs.msdn.microsoft.com/azuredev/?p=11235
  • https://techcommunity.microsoft.com/t5/Azure-Developer-Community-Blog/bg-p/AzureDevCommunityBlog
    • Since that platform is the new place to embrace the Microsoft developer and technical community, then that's where we need to be! All our blog posts are migrated over there, so you can still find the hits over there on our new site! Feel free to comment if you're looking for a particular post, but searching on the new blog will get you there. Also, this blog will be removed on 3/20, so we'll want to update any broken links to the new blog. Thank you all! We look forward to many more years as we dig into new technologies and content in Azure Development! - Ninja Ed]]>         Azure Content Spotlight - Azure Data Factory https://docs.microsoft.com/archive/blogs/azuredev/azure-content-spotlight-azure-data-factory-2 Thu, 14 Feb 2019 08:00:11 GMT https://blogs.msdn.microsoft.com/azuredev/?p=11215 relational, non-relational, and other storage systems. But raw data doesn't have the proper context or meaning to provide meaningful insights to analysts, data scientists, or business decision makers. Azure Data Factory provides a service that can orchestrate processes to refine these enormous stores of raw data into actionable business insights. It is build for complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects. To get started with Azure Data Factory, you can refer to the following video's and articles:   Cheers, Sjoukje ]]> Azure Content Spotlight - Azure Cost Management https://docs.microsoft.com/archive/blogs/azuredev/azure-content-spotlight-azure-cost-management Thu, 07 Feb 2019 21:17:18 GMT https://blogs.msdn.microsoft.com/azuredev/?p=11165 in Azure that could be more visible to the Azure community. With this week's announcement, Azure Cost Management now general availability for enterprise agreements and more!, our content spotlight is about Azure Cost Management. The Azure Cost Management has had several new features added so let's have a quick tour of some of its capabilities:

    Budgets:

    Budgets is a great feature for keeping an eye on costs and providing a mechanism for alerts on these budgets. The following screenshot shows a monthly budget of $100 and includes an alert for when the current running cost becomes 70% of the budget: Once the budgets have been defined, these are shown on the Budgets option as a convenient summary:

    Cost Analysis:

    Cost Analysis provides a means of tracking costs over a time period. The graph offers some great features including a time period, grouping and granularity. For example, the following graph shows daily consumption against the monthly budget and includes the different types of meter indicating that sql and event hubs are the most significant cost:

    References

    Both references have links to helpful videos: Lot's of great features to explore so have a look!]]>         Debugging Docker Containers (attaching) https://docs.microsoft.com/archive/blogs/azuredev/debugging-docker-containers-attaching Wed, 30 Jan 2019 20:00:19 GMT https://blogs.msdn.microsoft.com/azuredev/?p=11115 https://techcommunity.microsoft.com/t5/Azure-Developer-Community-Blog/Debugging-Docker-Containers-attaching/ba-p/338698 ================= Visual Studio has excellent built in support for working with Docker containers and most of the examples on the web are written with those tools installed. But what about the situation where the container is already running? This post will provide an example of debugging a .Net Framework application running a windows container as well as provide references to posts about remote debugging for different platforms including the .Net Framework, Python, and .Net Core applications.

    General

    In all three examples, the container needs to be prepared to allow for Visual Studio to access a running process on the container. Most devs are familiar with debugging solutions at design time using both the auto-attach (F5) and attaching the debugger to a running process on their local machine. This approach can be used to attach to a remote computer and even to Azure Web Apps and Azure Functions. Attaching to a container is similar to those examples where the ability to remote debug needs to be enabled on the container to allow Visual Studio the ability to connect.

    .Net Framework

    The first step to enabling the debug of a .Net Framework app is to install and run the remote debugger. The following is an example of enabling this on a dockerfile that uses Microsoft's Asp.Net image: [code lang="bash"] FROM microsoft/aspnet:10.0.14393.2665 EXPOSE 4020 4021 # download the remote debugger RUN Invoke-WebRequest -OutFile c:\rtools_setup_x64.exe -Uri https://download.microsoft.com/download/1/2/2/1225c23d-3599-48c9-a314-f7d631f43241/rtools_setup_x64.exe; # install the remote debugging RUN & 'c:\rtools_setup_x64.exe' /install /quiet # rest of the docker file [/code] After the image has been built, the next step is to make sure the ports for remote debugging (-p 4020:4020 -p 4021:4021) are also mapped when running the container. For example, the following starts the container with the name "mysite" and exposes ports 4020 and 4021 and port 80 as port 8078. [code] docker run -d -p 4020:4020 -p 4021:4021 -p 8078:80 --name mysite core [/code] The final step is to start the remote debugger on the container as shown below: [code] docker exec -it mysite "C:\Program Files\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger\x64\msvsmon.exe" /nostatus /silent /noauth /anyuser /nosecuritywarn [/code] Within Visual Studio, select the Attach to Process action in the Debug window: In the Transport, Remote is used and the Find button is used to establish the remote connection: The screenshot below shows the detected containers: If there is more than one container detected, you can determine the specific IP address of the container by using the docker inspect command: [code] docker inspect -f "{{ .NetworkSettings.Networks.nat.IPAddress }}" mysite [/code] Once the container has been selected, then the running process can be selected. For example, if debugging a IIS web application: Hope this helps! Please post any helpful comments for others and any other links you might know of!

    References

    .Net Framework

    Another example of remote debugging .Net Framework applications can be found in post by Richard Banks: How to Remote Debug a .NET App in a Windows Docker Container.

    .Net Core

    Offroad Debugging of .NET Core on Linux OSX from Visual Studio Debug .NetCore Containers in Docker and Kubernetes

    Python

    Python debug configurations in Visual Studio Code  ]]>         Azure Speech Service: Example project for custom voices, transcribe videos & generate subtitles! https://docs.microsoft.com/archive/blogs/azuredev/azure-speech-service-example-project-for-custom-voices-transcribe-videos-generate-subtitles Fri, 25 Jan 2019 15:25:35 GMT https://blogs.msdn.microsoft.com/azuredev/?p=11045 https://techcommunity.microsoft.com/t5/Azure-Developer-Community-Blog/Azure-Speech-Service-Example-project-for-custom-voices/ba-p/338692 ========================

    Feed audio from friends, family or vids of your favourite celebrity.

    Train a customised voice that sounds just like them.

    Then make them say whatever you want!

    [audio wav="https://msdnshared.blob.core.windows.net/media/2019/01/Martin-Fake-Voice-V2.wav"][/audio]   OK so this still sounds a little bit robotic. But the training data was less than 8 minutes of poor quality audio, with non-matching text. With not much more effort, you can buff this up to sound very close to the real person!   To hear what Martin sounds like (and learn a thing or two) watch this excellent series of clips about Azure DevOps.  

    Why implement a custom voice?

    This is a rapidly maturing technology, which has many real-world benefits. You can generate custom voices for chat bots, to use in games, for interacting with home automation, including Cortana, Alexa or Siri skills. It offers very believable video narration, and more realistic voice for call handling, to name just a few popular uses. The transcription service was where I came in. It is driven by the latest AI from Azure Cognitive Services. It provides almost real-time transcription and corrects itself as it goes.  

    Is it a lot of work to implement this new tech?

    As an IT pro reading the docs, training a "Custom Voice" with Microsoft Speech Service sounds like a complicated and cumbersome task. I also initially thought it sounded like too big of a job, to prepare all the training data for a decent sounding voice. But it wasn't, and I hope to dispel that myth for you as well! The services are very forgiving and will produce recognisable results with under a minute's worth of training data! The code I've uploaded to GitHub makes it even simpler.  

    What is a simplest way to implement this tech?

    Instead of spending hours reciting carefully scripted phrases for training data, why not flip it around? Take existing audio and first transcribe that to text. Then feed that the original audio and the auto-transcription BACK into the service! For even better results, spend as little as half an hour, tidying the transcription mistakes and feed that curated/cleaned version back in. You'll see how easy that is, with the demo tool I've posted posted to GitHub. It also helps you splice the audio up, into small chunks to feed into the service. This visual tool will do everything required to prep the training data and you can learn and adapt it to your needs. Read more and watch demos of this project HERE This project (which you can just download and run) will show you how to authenticate against the Speech Service, how to upload audio for transcription and also how to request and download the your final custom generated audio files.  

    Three simple steps to implement

    1. Transcribe your audio with the Speech to Text service
    2. Upload your snippets of curated phrases and text into the Custom Voice service
    3. Tell it what you want it to say and download the result
      You can download the code for this project from GitHub and have your own custom voice ready, in under an hour!   The initial results may initially sound a little "robotic", of course. But it's shocking to hear a familiar voice coming back at you. The final quality of your voice is actually only limited to the amount of time you then have. To tidy up the samples, perfect the acoustic library and work on the phonetic scripts. But you can still get very impressive results, with very little work.  

    Why did I need to implement this tech?

    Simply for better transcription results. I initially just wanted to provide viewers a transcript of a video presentation that Martin Woodward (Microsoft Principal Group Program Manager for Azure DevOps) gave, to my Azure Developer Meetup group last November.  You can watch the full set of eight clips from that great session here. The basic recognition code was copy n paste from the docs and I had transcription results within just FIVE minutes! However, the sound quality and his slight accent meant the transcription had many errors. they would have taken me some time to manually correct. So I decided instead to feed the "best bits" of the audio BACK into the service, generate an acoustic model and hopefully get a much improved transcription out. And it worked!   As a side-effect, I also generated a custom voice, which was trained to sound very similar to Martin! I was so impressed at how easy it was.   So over the festive break, I pulled together some demo code to show how it is implemented. This is now available in this C# GitHub repo, to demonstrate how to implement this.   You can download that now and try yourself. It's easy! Within just 30-60 minutes, you can get impressive results!   I'm filling out a more detailed article on using the service HERE on TechNet Wiki.   Plus, I'll be documenting the project features in the GitHub project wiki - if you just want to use the tool and not know the code.   Pete Laker, Azure MVP and Microsoft IT Implementer  
    This is part of a series of Azure developer blog posts, in my role as Microsoft IT Implementer. I'm honoured to have been asked by Microsoft to demystify and demonstrate some of the latest technologies, which excite me - and I hope you too!  ]]>         Azure Content Spotlight - Cognitive Services https://docs.microsoft.com/archive/blogs/azuredev/azure-content-spotlight-cognitive-services Thu, 24 Jan 2019 14:58:36 GMT https://blogs.msdn.microsoft.com/azuredev/?p=11025 https://techcommunity.microsoft.com/t5/Azure-Developer-Community-Blog/Azure-Content-Spotlight-Cognitive-Services/ba-p/338688 ======================== Welcome to another Azure Content Spotlight! These articles are used to highlight items in  Azure that could be more visible to the Azure community. This week's spotlight highlights Cognitive Services containers. These containers offer a mechanism for taking Azure Cognitive Services to the IoT Edge as well as running behind infrastructure that might prevent external network activity. In This Week in Cognitive, Noelle LaCharite provides a great summary of the state of these services and Microsoft's motivation for providing them for their customers. Additionally, Christoffer Noring provides an entertaining introduction to using the Cognitive Services API in Cognitive Services API — I need your clothes, boots and your motorcycle including examples using the Azure CLI to create the service and using curl to call the services.      ]]> Azure Content Spotlight – Azure Data Architecture Guide https://docs.microsoft.com/archive/blogs/azuredev/azure-content-spotlight-azure-data-architecture-guide Thu, 17 Jan 2019 03:12:25 GMT https://blogs.msdn.microsoft.com/azuredev/?p=11005 https://techcommunity.microsoft.com/t5/Azure-Developer-Community-Blog/Azure-Content-Spotlight-8211-Azure-Data-Architecture-Guide/ba-p/338686 ======================== Welcome to another Azure Content Spotlight! These articles are used to highlight items in Azure that could be more visible to the Azure community. This week's content spotlight is about the Azure Data Architecture Guide, which is published by the Azure CAT team. This guide can be used by data professionals, architects and developers, who are interested in selecting the right data services and technologies on Azure. It will give you a lot of interesting information, and guidance on how to design effective data architectures and solutions on the Azure platform. The Azure CAT team provided a series of blog posts:   You can also refer to the Azure Data Architecture guide.   Cheers, Sjoukje
    to use Ginger
    Limited mode
    the Azure Data Architecture guide
    ×
    ]]>         Playing with Containers in Azure App Service https://docs.microsoft.com/archive/blogs/azuredev/playing-with-containers-in-azure-app-service Mon, 14 Jan 2019 09:00:37 GMT https://blogs.msdn.microsoft.com/azuredev/?p=10875 https://blogs.msdn.microsoft.com/azuredev/2019/01/07/setting-up-a-docker-containers-infrastructure-with-azure-devops/) You might be able to grasp the beauty of packaging code to Docker images and then deploy a number of containers, but you still need somewhere to actually run these containers. The first choice of this author is usually Azure, but which service in Azure? You have more than one way to skin the proverbial cat in Azure. I'm not going to cover all these options within this post, but will instead focus on Azure App Service for now. Turns out there are a couple of options inside that offering as well. Let's cover a basic scenario. We have a setup where we create a frontend and a backend, and want them to communicate with each other. I'm leaving the Docker Hub deployment out of this post - I'll show the Azure parts and assume the images are retrievable. Start by creating an empty solution in Visual Studio: Then add a web app: Add a Dockerfile.CI (if you want to pipe it through Azure DevOps): [code language="csharp"] FROM microsoft/dotnet:2.2-sdk AS build-env WORKDIR /app # Copy csproj and restore as distinct layers COPY *.csproj ./ RUN dotnet restore # Copy everything else and build COPY . ./ RUN dotnet publish -c Release -o out # Build runtime image FROM microsoft/dotnet:2.2-aspnetcore-runtime WORKDIR /app COPY --from=build-env /app/out . ENTRYPOINT ["dotnet", "FooFrontend.dll"] [/code] After testing and building locally you should make sure it ends up as an image you can retrieve. For the purposes of this example I have pushed it to https://hub.docker.com/r/ahelland/hellofoo-frontend. Create a web app in Azure: You will get three options - let's start with Single Container: Startup file should match your ENTRYPOINT in your Dockerfile. Open the web app to verify it worked: That was just pure boilerplate code though. Let's add some text like "Hello from Docker". For instance we could edit index.cshtml to contain a div like this: [code language="csharp"] <div class="text-center"> Hello from Docker!<br /> </div> [/code] That's fairly static though, and we want this to be configurable. Using the standard mechanisms in .NET Core we create a setting in appsettings.json: [code language="csharp"] { "Logging": { "LogLevel": { "Default": "Warning" } }, "AllowedHosts": "*", "UI_Texts": { "Greeting": "Hello from Docker!", } } [/code] And edit index.cshtml accordingly: [code language="csharp"] @page @using Microsoft.Extensions.Configuration; @inject IConfiguration configuration; @model IndexModel @{ ViewData["Title"] = "Home page"; } <div class="text-center"> Greeting: @configuration.GetSection("UI_Texts")["Greeting"]<br /> </div> [/code] This works, but it persisted in the Docker image - what if we want to change it? And what if this was just a placeholder value specifically because we don't want to embed secrets in the image? If we go back to the cloud (after checking in the code and pushing to Docker Hub) we can add an application setting in Azure App Service: Do note that the hierarchical definition is different for containers than non-containerized app service instances. The format for non-container is parent.child whereas containers require parent__child (double underscore). This modifies the environment settings in for the host, and Docker picks up on this. A quick refresh of the browser and it should have changed (no redeploy necessary): That's quite a nice way to handle configuration. No touching of the file system necessary. I initially said that I wanted to do a frontend vs backend setup, but this just covered a frontend. The next step would be adding a backend: And we need a Dockerfile.CI for this project too. [code language="csharp"] FROM microsoft/dotnet:2.2-sdk AS build-env WORKDIR /app # Copy csproj and restore as distinct layers COPY *.csproj ./ RUN dotnet restore # Copy everything else and build COPY . ./ RUN dotnet publish -c Release -o out # Build runtime image FROM microsoft/dotnet:2.2-aspnetcore-runtime WORKDIR /app COPY --from=build-env /app/out . ENTRYPOINT ["dotnet", "FooBackend.dll"] [/code] While this API doesn't do much we would still like to print out the result of an API-call. Let's use the same approach as before - add a setting to appsettings.json: [code language="csharp"] "UI_Texts": { "Greeting": "Hello from Docker in the cloud!" }, "API" { "api_address": "https://localhost:1234/api/values" } [/code] And add it to index.cshtml: [code language="csharp"] @page @using Microsoft.Extensions.Configuration; @using System.Net.Http; @inject IConfiguration configuration; @model IndexModel @{ ViewData["Title"] = "Home page"; } @{ using (var client = new HttpClient()) { var apiAddress = configuration.GetSection("API")["api_address"]; string requestUrl = apiAddress; HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, requestUrl); HttpResponseMessage response = client.SendAsync(request).Result; var responseString = response.Content.ReadAsStringAsync().Result; ViewBag.result = responseString; } } <div class="text-center"> Greeting: @configuration.GetSection("UI_Texts")["Greeting"]<br /> Result of API call: @ViewBag.result; </div> [/code] You'll notice I needed to add some inline code to execute the http call, and then I pass it back via the ViewBag. Compile, test, check in and all that jazz. I posted this image to: https://hub.docker.com/r/ahelland/hellofoo-backend We'll want to move back to the Azure Portal to sort this out, but there are a couple questions to be answered first. We used "Single container" in the first round, and this doesn't sound like one container? True, and that's why there is an option for Docker Compose, as the purpose of that is specifically setting up more than one container and in some cases a relationship between containers. Can we still use the application settings in the portal to set the config for Docker-compose? No, we can't. But is is possible to do via environment settings in the Docker config. (See below.) Well, what do I use for the address - clearly "localhost" doesn't sound right? And here is another beauty about containers; you just refer to it by name and let Docker handle the rest. (Also see below.) [code language="csharp"] version: '3.4' services: foofrontend: image: ahelland/hellofoo-frontend:latest environment: - UI_Texts__Greeting="Hello from Docker-Compose" - API__api_address=https://foobackend/api/values - ASPNETCORE_URLS=https://+:80 ports: - "59372:80" foobackend: image: ahelland/hellofoo-backend:latest environment: - ASPNETCORE_URLS=https://+:80 ports: - "59376:80" [/code] This might actutally take a minute or two to fall in place, but if everything went as planned the web page should look roughly like this: I have noticed that there are sometimes hangups in the UI where you need to refresh to make a new Docker-Compose "stick", but that's probably a preview thing. This approach works the way we want it for our little setup, but be aware that there are some limitation to this: - You can only expose one container on the internet; in this case the frontend (it comes first in the Docker-Compose file). The API is only available inside the container network. - If you want/need fine-grained control of ports you will notice that Azure App Service does some magic behind the scenes for you which may or may not be ok for you. - Troubleshooting when it doesn't work isn't all that easy by default, meaning that you might want to make sure you log extra info to the console (which should be visible in the portal) If you want/need more complicated container setups it might be that Azure App Service isn't right for you. Maybe you need Kubernetes or something. That is outside the scope of this post - this post is just to show some of the thing a web apps can do for you.]]> Setting up a Docker Containers Infrastructure with Azure DevOps https://docs.microsoft.com/archive/blogs/azuredev/setting-up-a-docker-containers-infrastructure-with-azure-devops Mon, 07 Jan 2019 09:00:27 GMT https://blogs.msdn.microsoft.com/azuredev/?p=10725 HelloDocker web app. Afterwards I added Docker support for Linux containers. The default Dockerfile isn't perfect for Azure DevOps pipelines though so I created one for that purpose called Dockerfile.CI: [code language="csharp"] FROM microsoft/dotnet:2.2-sdk AS build-env WORKDIR /app # Copy csproj and restore COPY *.csproj ./ RUN dotnet restore # Copy everything else and build COPY . ./ RUN dotnet publish -c Release -o out # Build runtime image FROM microsoft/dotnet:2.2-aspnetcore-runtime WORKDIR /app COPY --from=build-env /app/out . ENTRYPOINT ["dotnet", "HelloDocker.dll"] [/code] Next step is pushing it to GitHub - I'll assume you've got that part covered as well. I have also made sure I have a Docker Hub id, and the credentials for that ready. So, let's move into Azure DevOps and create a project: Then you will want to navigate to the Project Settings, and Service connections for Pipelines: First let`s add a GitHub repo - you have two choices: Use External Git: Use GitHub: External Git will let you add any random repo on GitHub, whereas the GitHub connection requires you to have permissions to the GitHub account. (The benefits are that you can browse the repo instead of typing in paths manually, you can report back to GitHub that releases should be bundled up, etc. so provided it is your repo you should consider using a GitHub connection.) That means you have to login and consent for the GitHub connection: Since we're already in the service connection let's add a Docker registry as well: Then we will move on to creating a pipeline. If you chose an External Git repo choose that as a source: Or GitHub as a source if that's what you set up in the previous step - that requires you to choose the repository you want to work with: There are a number of templates to choose from, but to make it easy you can opt for the Docker container template: I chose to do builds both for ACR and Docker hub in the same pipeline by adding two more Docker tasks. (For simple customization needs this might not be required, but options are always great.) The build task looks like this: And pushing the image: Kick off the build, and hopefully you will get something along these lines: Which means you should also have an image available for everyone: And there you have it - pull from Docker Hub into Azure Web Apps, do a replace a number of files and push a customized image into ACR only to pull into AKS afterwards. Well, you catch my drift :)]]> Enforcing security controls right from CI/CD pipeline with AzSK - Deep Dive https://docs.microsoft.com/archive/blogs/azuredev/enforcing-security-controls-right-from-ci-cd-pipeline-with-azsk-deep-dive Sat, 29 Dec 2018 08:17:15 GMT https://blogs.msdn.microsoft.com/azuredev/?p=10655
  • The attestation module allowing for a full traceability of security controls deviation and justification of why a given control was not respected, which may be very useful in case of internal/external audit
  • The CI/CD extensions available on the marketplace. This makes possible to enforce security controls as from CI builds, so very early in the application lifecycle. On top of Azure DevOps extensions, the kit also ships with Visual Studio extensions to provide security guidance as the developer is typing the code.
  • An ARM Template Security checker that is available from CI/CD as well as separate cmdlet. This ensures ARM templates conform to security policies defined at organization level.
  • A lot of room for customizing control settings, creating organization specific controls, etc. It is free, the only costs you might have would be incurred by a few resources (Storage Account & Automation) that are required to use the kit but overall, it is very low.
    • The kit comes with other things such as Continuous Assurance (CA) & integration with OMS but that's where there is a bit of overlapping with Azure Security Center (ASC). The latter being much easier to use but less customizable and does not cover CI/CD. AzSK's documentation is rather good but it still takes a bit of time to get acquainted, especially in the CI/CD domain as this is still a work in progress. I dogged into the source code of AzSK and more particularly into the ARM Library Checker (.NET & .Net core module) in order to understand exactly how it works and that's what I'm going to share here. That said, you should already know how to create a custom org policy and setup CA, etc. as I'm not going to go through all the details. I guess the official doc all in all is certainly about 50 pages. If you don't know AzSK at all, you may still read this article but it's going to be harder to grasp it correctly.

    Step 1 - working with the Azure DevOps tasks

    Currently, the extension comes with 2 tasks, the AzSK_ARMTemplateChecker used to inspect and validate ARM template files and AzSK_SVTs, used to validate the overall security of the resource group that is bound to the current release or use a tagname/value pair that identifies resources used and deployed by your application. AzSK_ARMTemplateChecker  is meant to be used as a pre-deployment scan while AzSK_SVTs is meant to be used as a post-deployment scan. Even when you use AzSK_ARMTemplateChecker, it is a good idea to also validate the overall solution with AzSK_SVTs as you might have tasks in your release pipeline that could modify some resources not via ARM templates but rather via Powershell, Azure Cli, etc., making a final scan with AzSK_SVTs necessary. Those two tasks are using cmdlets behind the scenes which can also be used directly by sys admins. The configuration of the tasks is explained in the AzSK doc but I'd like to come with extra details that are not discussed in the official doc. For the time being, the ARM checker task does not work with parameters which means that it expects to have the actual parameter values as part of the template. So, for instance, one of the built-in controls checks how many instances of a given app service plan runs. The control doing this is the following: [code langage="json"] { "id": "AppService270", "apiVersions": [ "2016-09-01" ], "controlId": "Azure_AppService_BCDR_Use_Multiple_Instances", "isEnabled": true, "description": "App Service must be deplsoyed on a minimum of 1 instance to ensure availability", "rationale": "App Service deployed on multiple instances ensures that the App Service remains available even if an instance is down.", "recommendation": "Run command 'Set-AzureRmAppServicePlan -Name '&amp;amp;amp;lt;AppServicePlanName&amp;amp;amp;gt;' -ResourceGroupName '&amp;amp;amp;lt;RGName&amp;amp;amp;gt;' -NumberofWorkers '&amp;amp;amp;lt;NumberofInstances&amp;amp;amp;gt;''. Run 'Get-Help Set-AzureRmAppServicePlan -full' for more help.", "severity": "High", "jsonPath": [ "$.sku.capacity" ], "matchType": "IntegerValue", "data": { "type": "GreaterThan", "value": 1 } } [/code] as you can see, the jsonPath attribute contains the relative path to the property of the corresponding resource in the controlled ARM template, which is typically something like this: [code langage="json"] "resources": [ { ... "sku": { "name": "[parameters('skuName')]", "capacity": "[parameters('skuCapacity')]" }, ... } [/code] where the value of "capacity" is supposed to come from the "skuCapacity" parameter. However, if you ask the ARM checker task (as well as the cmdlet) to check the template "as is", it will fail because the condition in the control states that the value should be greater than 1, meaning that the library will compare "[parameters('skuCapacity')]" > 1 which will of course fail. Since hardcoding values directly in the template isn't CI/CD friendly, the easiest way to workaround this is to simply make a configuration transformation step as follows: where you can specify which transformation to apply. In this case, I'm asking the task to take my first resource and replace the value of the sku capacity by the value of my release variable, and that will do the trick. For sake of simplicity, I'm illustrating this with only this parameter but you should of course do that with all the parameters. Next you can simply use the AzSK_ARMTemplateChecker task that is easy to configure. Now, I had hard times finding how to specify which controls should to be taken into account by the task. Although the official documentation is quite exhaustive, there are still some grey zones and I had hard times making it work because there is currently (MS is working on that) a bug (specific to ARM checker) with this feature and that's why it didn't work for me from the start.

    Step 2 - controls being checked by the ARM Checker

    By default, if you haven't setup a custom organization policy, the controls that will be checked by AzSK are retrieved from the default online policy store: https://azsdkossep.azureedge.net/1.0.0/ARMControls.json . The file contains a series of resources which in turn contain a series of controls. The control I showed earlier (AppService270) is part of this file under the AppService resource. So, by default, all controls that are applicable to resources that are part of your ARM template will be executed. If you want to use your own controls, you first need to use the latest version of AzSK (currently 3.9.0), then you have to take a copy of ARMControls.json, modify it as you wish and upload it to your own online policy store. That's what I did but still, I thought it didn't work because of the bug I was referring to earlier: the control's isEnabled attribute is currently ignored by the ARM Checker (but works fine with SVT...) so I thought the system was not picking up my control file. Then I jumped to my self-hosted build agent and started debugging and troubleshooting with Fiddler and I noticed that my file was well retrieved. So, I ended up cloning the AzSK repo and made a step by step debugging of the ARM checker library and realized that this property was just ignored. So, I fixed that in my own local environment and MS should release a fix ASAP. Once you uploaded your own version of ARMControls.json, you also have to specify how you want to use it in ServerConfigMetadata.json, and this is either like this: [code langage="json"] { "Name" : "ARMControls.json", "OverrideOffline":true } [/code] either like this: [code langage="json"] { "Name" : "ARMControls.json"} [/code] With the first option, you tell AzSK that you want to use only your file. The second option tells AzSK to overlay the out of the box ARMControls with yours which allows you to only specify the delta. Up to you to decide which one you prefer. Then, you may change existing controls and even add your own. If we come back to the control I showed earlier: [code langage="json"] { "id": "AppService270", "apiVersions": [ "2016-09-01" ], "controlId": "Azure_AppService_BCDR_Use_Multiple_Instances", "isEnabled": true, "description": "App Service must be deplsoyed on a minimum of 1 instance to ensure availability", "rationale": "App Service deployed on multiple instances ...", "recommendation": "Run command 'Set-AzureRmAppServicePlan...", "severity": "High", "jsonPath": [ "$.sku.capacity" ], "matchType": "IntegerValue", "data": { "type": "GreaterThan", "value": 1 } } [/code] we could enable it or disable it (again not working right now), decide that GreaterThan 0 would be more suitable for a development environment as you don't especially want to pay for multiple instances in dev, change the description, severity etc. So, except the JsonPath attribute that you should never change, you can do whatever you want with the rest. If you want to add your own controls, as for instance enforcing MSI or making sure ARR Affinity is disabled, you could simply add the following JSON body under the AppService resource: [code langage="json"] { "controlID": "Azure_AppService_ClientAffinity_SEY_0001", "description": "ARR Affinity must be turned off for App Service;", "id": "AppService_Client_Affinity_SEY_0001", "severity": "Medium", "recommendation": "Go to Azure Portal --&amp;amp;gt; your App Service --&amp;amp;gt; Settings --&amp;amp;gt; Application Settings --&amp;amp;gt; ARR Affinity --&amp;amp;gt; Click on 'OFF'.", "jsonPath": [ "$.properties.clientAffinityEnabled" ], "matchType": "Boolean", "data": { "value": true }, "isEnabled": false, "rationale": "ARR Affinity makes it easier to drive DOS/DDOS attacks against the underlying resource" } [/code] This control is not enforced by AzSK right now but I personally find ARR Affinity may pose a security risk since when turned on, the webapp will return a cookie named ARRAffinity containing the specific backend instance ID to which Azure should target client requests. That makes it easy for attackers to overload always the same instance until it is down and then get a fresh ARRCookie etc. until all the instances are down. In any case, ARR Affinity is often required by poorly written applications that were not correctly designed to run on multiple nodes. So, the earlier such issues are discovered (in the CI pipeline), the better!

    Step 3 - controls being checked by the SVT task

    As a reminder, the SVT task is to be used in a post-deployment phase so for instance, it could be the last task of a release pipeline. SVT works differently than ARM Checker although the ultimate purpose is the same. For SVT, all the control files live in the [AzSK Module Path]\[AzSK version]\Framework\Configurations\SVT\Services and each resource has its own corresponding JSON file. If we look into AppService.json, we'll find a similar control to the one we worked with so far: [code langage="json"] { "ControlID": "Azure_AppService_BCDR_Use_Multiple_Instances", "Description": "App Service must be deployed on a minimum of two instances to ensure availability", "Id": "AppService270", "ControlSeverity": "Medium", "Automated": "Yes", "MethodName": "CheckAppServiceInstanceCount", "Rationale": "App Service deployed on&amp;amp;nbsp;....", "Recommendation": "Run command ...", "Tags": [ "SDL", "TCP", "Automated", "BCDR", "AppService" ], "Enabled": true, "FixControl": { "FixMethodName": "SetMultipleInstances", "FixControlImpact": "Medium" } } [/code] but as you might have noticed, there is no JsonPath attribute nor any rule to define the behavior of the control. Instead, there is the MethodName attribute which indicates which PowerShell method should be called when this control is being executed.  In this case, the method CheckAppServiceInstanceCount of the resource AppService should be called but where is the code defined? Well, simple: in the AppService.ps1  ([AzSK Module Path]\[AzSK version]\Framework\Core\SVT\Services) script that defines the AppService class and sure enough, if you look for CheckAppServiceInstanceCount into that file, you'll find its implementation. That said, you should *never* change any of the local files. For SVT controls, you can change some settings in the ControlSettings.json file but if you want to change the behavior or a given control, or add a new control, you'll have to extend the resource. For all the details, you should read the official doc but I'm going to show you how I extended SVT to create the ARR Affinity check in a similar way as for the ARM checker. Since ClientAffinityEnabled (displayed as ARR Affinity in the portal) is a property of an Azure webapp, I extended the AppService feature by adding two new files to my org policy folder: AppService.ext.json that contains the declarative definition of the control AppService.ext.ps1 that contains the method to call when the control executes The contents of AppService.ext.json are: [code langage="json"] { "FeatureName": "AppService", "Reference": "aka.ms/azsktcp/appservice", "IsMaintenanceMode": false, "Controls": [ { "ControlID": "Azure_AppService_ClientAffinity_SEY_0001", "Description": "ARR Affinity must be turned off for App Service", "Id": "AppService_Client_Affinity_SEY_0001", "ControlSeverity": "Medium", "Automated": "Yes", "MethodName": "CheckAppServiceClientAffinityEnabled", "Rationale": "ARR Affinity makes it easier to drive DOS/DDOS attacks against the underlying resource", "Recommendation": "Go to Azure Portal ...", "Tags": [ "SDL", "TCP", "Automated", "Config", "AppService", "FunctionApp" ], "Enabled": true } ] } [/code] where the most important part is the method name CheckAppServiceClientAffinityEnabled that is implemented in AppService.ext.ps1: [code langage="PowerShell"] #using namespace Microsoft.Azure.Commands.AppService.Models Set-StrictMode -Version Latest class AppServiceExt: AppService { hidden [PSObject] $ResourceObject; hidden [PSObject] $WebAppDetails; hidden [PSObject] $AuthenticationSettings; hidden [bool] $IsReaderRole; AppServiceExt([string] $subscriptionId, [string] $resourceGroupName, [string] $resourceName): Base($subscriptionId, $resourceGroupName, $resourceName) { $this.GetResourceObject(); $this.AddResourceMetadata($this.ResourceObject.Properties) } AppServiceExt([string] $subscriptionId, [SVTResource] $svtResource): Base($subscriptionId, $svtResource) { $this.GetResourceObject(); $this.AddResourceMetadata($this.ResourceObject.Properties) } hidden [ControlResult] CheckAppServiceClientAffinityEnabled([ControlResult] $controlResult) { if([Helpers]::CheckMember($this.WebAppDetails,"ClientAffinityEnabled")) { if($this.WebAppDetails.ClientAffinityEnabled) { $controlResult.AddMessage([VerificationResult]::Failed, [MessageData]::new("ARR Affinity for resource " + $this.ResourceContext.ResourceName + " is turned on", ($this.WebAppDetails | Select-Object ClientAffinityEnabled))); } else { $controlResult.AddMessage([VerificationResult]::Passed, [MessageData]::new("ARR Affinity for resource " + $this.ResourceContext.ResourceName + " is turned off", ($this.WebAppDetails | Select-Object ClientAffinityEnabled))); } } else { $controlResult.AddMessage([VerificationResult]::Manual, [MessageData]::new("Could not validate ARR Affinity settings on the AppService: " + $this.ResourceContext.ResourceName + ".")); } return $controlResult; } }[/code] in which I simply check the value of the ClientAffinityEnabled webapp property and I return the control result (Passed, Failed, Manual) accordingly. Once you've uploaded your extension files to your org policy storage account, you also have to list them in ServerConfigMetadata.json.

    Step 4 - different policies for different environments?

    Depending on the control baseline (which I'm not covering in this article) you want to define, you might decide to use a single corporate policy for all your subscriptions and environments but you might also want to define different policies per environment. For instance, with the example I talked about so far (multiple instances enforced by the control), you might want the control to behave differently whether you are in dev (where a single instance is enough) or in staging/prod where you should indeed have more than 1 instance to be HA. No matter what you decide and independently of AzSK, the way your environments are organised (1 vs multiple subscriptions) will also influence the way you'll tackle your policies. AzSK supports to have either one policy store per subscription, either one global store. However, I noticed that this doesn't matter too much, especially when it comes to CI/CD as the underlying tasks will trigger a fresh PowerShell window for every execution and target a given policy store using the Set-AzSKPolicySettings cmdlet. So, if you decide to work with only one policy store cross-environments (DEV/STAGING/PROD), you might simply define your build/release variables this way: scoped at release level and then let your different stages use the same settings. However, should you use different policy stores, you may define them like this: in this case I defined AzSKServerUrl twice with different values and scoped respectively to the DEV & STAGING stages. My org policy storage account looks like this: So, I simply created one container per environment and each container contains its own control files: Hope this helps!]]>