Gabe Shapiro's blog

Dynamically pass a SAS token into your ARM template via PowerShell

Gabe Shapiro — Thu, 12 Jul 2018 18:12:17 GMT

There are scenarios where you may need to pass an Azure Storage Shared Access Signature (SAS) token into your Azure Resource Manager (ARM) template. For example, you might want to deploy an App Service with Application and/or Web Server diagnostic logs set to log to Blob Storage, or you might want to deploy site content as a blob to App Service via the MSDeploy ARM extension. Since your blobs might contain sensitive information, storing the SAS token in your template file or even in your parameters file can pose a security risk. Nor do you want to manually enter the sas token during deployment because this would defeat the purpose of automated deployment. Instead, you can dynamically generate the sas token and then pass it in as a secure string parameter when you run your ARM deployment script. Below is an example of how to do this with ARM PowerShell. These steps assume that you are already authenticated with your Azure account (ie your Service Principal) that has authorization to get Storage keys & deploy your resources, Essentially, the process is as follows:

Get one of the Storage keys via Get-AzureRmStorageAccountKey .
Generate a SAS token (in this example, via New-AzureStorageContainerSASToken ).
Convert the SAS token to a Secure String. For simplicity in the below example, I converted the entire constructed SasURL (which contains the SAS token) to Secure String. The reason we convert to Secure String is to conceal the value during deployment.
Pass the SAS token (or in this example, the SAS URL for simplicity) into New-AzureRmResourceGroupDeployment as a parameter (separate from your parameters file).
In your ARM template, create a parameter with a name that matches the name of the secure string parameter you are passing into New-AzureRmResourceGroupDeployment. In your ARM template, make sure to set this parameter type as SecureString.


$TemplateFile = "C:\myfakepath\template.json";
$TemplateParameterFile = "C:\myfakepath\template.parameters.json";
$ResourceGroupName = "MyFakeResourceGroup";
$storageAccountName = "MyFakeStorageAccount";
$Container = "myfakecontainer";
<# In this example we are using the primary Storage key to generate the sas token, but you can use the secondary Storage key if you choose to #>

$storageKey0 = (Get-AzureRmStorageAccountKey -AccountName $storageAccountName -ResourceGroupName $ResourceGroupName).Value[0];
$context = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageKey0;
$sasTokenPlain = New-AzureStorageContainerSASToken -Context $context -Container $Container -Permission "rwd";
$SasUrlPlain = "https://$storageAccountName.blob.core.windows.net/$Container$sasTokenPlain";

<# converting the sasURL to secure string. Be sure to define the sasUrl parameter in your template as a SecureString #>

$SasUrl = ConvertTo-SecureString $SasUrlPlain -AsPlainText -Force;

<#

For more info about New-AzureStorageContainerSASToken, see the following documentation:

https://docs.microsoft.com/en-us/powershell/module/azure.storage/new-azurestoragecontainersastoken?view=azurermps-6.4.0

#>

 

$OptionalParameters = New-Object -TypeName Hashtable;

$OptionalParameters['sasUrl'] = $SasUrl;

$DeploymentName = "myfakedeployment";

New-AzureRmResourceGroupDeployment -Name $DeploymentName -ResourceGroupName $ResourceGroupName -TemplateFile $TemplateFile -TemplateParameterFile $TemplateParameterFile @OptionalParameters;

Abbreviated template snippet:



"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

"contentVersion": "1.0.0.0",

"parameters": {

"siteName": {

"type": "String"

},

"sasUrl": {

"type": "SecureString"

}

}

"resources": [

{

"type": "Microsoft.Web/sites/config",

"name": "[concat(parameters('siteName'), '/logs')]",

"httpLogs": {

"azureBlobStorage": {

"sasUrl": "[parameters('sasUrl')]",

"retentionInDays": 1,

"enabled": true

}

},

PowerShell script to save a list of folder and file contents & file sizes to a csv file in App Service

Gabe Shapiro — Sat, 02 Sep 2017 16:13:17 GMT

When you encounter an unexpected runtime symptom that occurs after you deploy to your site, there are situations where it makes sense to further isolate whether this is a runtime issue or a deployment issue. A deployment issue is an issue where the wrong files are deployed. A runtime issue is an issue where the expected files are deployed, but the application doesn't work as expected. For example, such situations might include ones where the (presumably) same deployment works on another site, or where the site doesn't render as expected, or where you get a consistent error doesn't seem to have any relation to the change that was made to the application, or a consistent runtime error in a new Kudu deployment on a new App Service that doesn't reproduce when deployed to a different environment (eg on-premises IIS).' The simplest way to determine whether the correct files are deployed are to compare the contents of the App Service's wwwroot folder to the contents of a working site to see if they match. Here is a PowerShell script that lists the contents of the wwwroot folder in App Service and saves this list to a csv file. It saves the list of contents to d:\home\site\temp\contents.csv . You can also use the script to list the contents of the local app (change the $siteFolder and $tempFolder values to point to appropriate locations on your local app's server). You can then use a content comparison tool (MS Word, for example) to compare the contents of the local app's folder and App Service folder. You can use the App Service's PowerShell Kudu console to run the script in App Service, as well as to download the csv that gets generated: https://.scm.azurewebsites.net/DebugConsole/?shell=powershell <# folder to recursively list the contents of #>

$siteFolder="D:\home\site\wwwroot";

<# path to write the list of contents to #>

$tempFolder="d:\home\site\temp";

<# csv file to write the list of contents to #>

$csvFile="contents.csv";

<# Delimiter to use in the csv file #>

$delimiter= ',';

if (!(Test-Path $tempFolder -PathType Container)) {New-Item -ItemType Directory -Force -Path $tempFolder}get-childitem $siteFolder -rec | select-object FullName, Length | export-csv -notypeinformation -delimiter $delimiter -path $tempFolder\$csvFile

Further information about investigating deployment versus runtime issues is posted here: https://github.com/projectkudu/kudu/wiki/Deployment-vs-runtime-issues

Redirect HTTP to HTTPS when using ASP.NET Core in Web App on Linux - Azure App Service

Gabe Shapiro — Mon, 14 Aug 2017 21:33:08 GMT

A couple things to consider when implementing HTTP to HTTPS redirect for an ASP.NET Core application running in Web App on Linux - Azure App Service. -Since you are using the .NET Core server to process requests, configuring a redirect rule in .htaccess won't work the way it does with sites that are closely coupled with Apache. What you can do instead is implement the redirect in code, using ASP.NET Core Middleware to handle the request and response. -SSL is offloaded to port 80 from the ARR / front end servers, so using a method such as AddRedirectToHttps() causes the request to end up in an infinite redirect loop since the offloaded requests remain on port 80. The way around this is to check for the presence of an X-ARR-SSL header, which indicates that the originating request on the ARR front end was sent over SSL and if this header is not present, redirect to https://url . Here is a basic example of how to implement this redirect via middleware: 1. Create a Middleware class that contains the logic to redirect the request:

using Microsoft.AspNetCore.Http;

namespace MyNamespace { public class RedirectToHttpsArrMiddleware { private readonly RequestDelegate _next;

public RedirectToHttpsArrMiddleware(RequestDelegate next) { _next = next; }

public Task Invoke(HttpContext context) { //if the request isn't HTTPS if (!(context.Request.IsHttps || context.Request.Headers.ContainsKey("X-ARR-SSL"))) {

//redirect to https://url var request = context.Request; var redirectUrl = $"https://{request.Host}{request.PathBase}{request.Path}{request.QueryString}"; context.Response.Redirect(redirectUrl, true);

} // Call the next delegate/middleware in the pipeline return this._next(context); } }

2. Create an extension class that will the Middleware via IApplicationBuilder:

using Microsoft.AspNetCore.Builder;

namespace MyNamespace { public static class RedirectToHttpsArrMiddlewareExtensions { public static IApplicationBuilder UseRedirectToHttpsArr( this IApplicationBuilder builder) { return builder.UseMiddleware(); } } }

3. In the Configure method of the Startup class (in Startup.cs), invoke the Middleware. public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory) { loggerFactory.AddConsole(); if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); } app.UseRedirectToHttpsArr(); app.Run(async (context) => { await context.Response.WriteAsync( $"Hello world"); }); } Further information about using ASP.NET Core Middleware is found at the following link: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware

Automatically stop (and start) an Azure App Service main site, or disable and re-enable Always On, during Git- / Kudu-based deployment

Gabe Shapiro — Sun, 13 Aug 2017 23:02:52 GMT

Let's say you do GitHub or Bitbucket continuous deployments to an Azure App Service. At some point, deployments start failing, and after a bit of further investigation, you notice in the Kudu deployment logs that a file in use or permissions error is occurring when trying to copy over existing files of the same names. Now let's say that temporarily stopping the main site, or that disabling Always On, allows you to redeploy successfully. So you have a workaround. While not a showstopper, you would prefer to not have to manually intervene in your continuous deployments. There are some tricks you might be able to implement with regard to specific situations (for example renaming certain files on the site or skipping copying these files during deployment). However, since the inclusion of Azure Resource Manager (ARM) PowerShell modules on the App Service image, you can now run ARM PowerShell commands from your App Service, which means that can do operations such as stopping and starting your main site process by using ARM commands. You can incorporate these commands into a Kudu custom deployment script (deploy.cmd) to automatically stop the site towards the beginning of the Kudu deployment script, and start the site at the end of the Kudu deployment script. There are a couple limitations to this approach that I should mention: -This approach will make your site unavailable while the deployment is occurring. However, you can work around this by using deployment slots, and automate the process of swapping the slots via AutoSwap (or alternatively via an additional ARM call). -This approach currently relies upon PowerShell (which can also be executed from the Command-Line as I am doing in this example). AFAIK there isn't a way to make this work if using a bash script within App Service (deploy.sh). Here are basic steps to use this approach. You will of course want to test this piece-by-piece to ensure that each part works and to avoid causing issues in production.

To run the ARM PowerShell commands in App Service, you will need a Service Principal. I have posted basic steps on how to create a Service Principal in preparation here.
To invoke the PowerShell commands during deployment, you will need a deployment script. You can download the deploy.cmd that gets generated when you do a Kudu deployment, by going to https://.scm.azurewebsites.net/DebugConsole , navigating to D:\home\site\deployments\tools and then downloading the deploy.cmd via the download icon. Alternatively, you can generate a custom deploy.cmd by using the steps described here .
To use the deployment script, you will need to add a .deployment file to the root of your repository (note that there is a dot (.) at the beginning of the filename), and point this configuration to the deploy.cmd as follows: [config] command = deploy.cmd
Edit the deploy.cmd so that it stops the site at the beginning of deployment and starts the site after the deployment is completed:

To stop the site at the beginning of the deployment: place the following command near the top of the file (after the @if "%SCM_TRACE_LEVEL%" NEQ "4" @echo off line), replacing the values in single quotes (ie your service principal user name and password, tenant id, App Service's resource group, and App Service name): powershell -command "Login-AzureRmAccount -Credential (New-Object -TypeName pscredential –ArgumentList 'appId@domain.com', (ConvertTo-SecureString 'password' -AsPlainText –Force)) -ServicePrincipal –TenantId 'tenantId';Stop-AzureRmWebApp -ResourceGroupName 'resourcegroup' -Name 'mysite';"

To start the site at the end of the deployment: Place the following command at the end of the file (same as the above command, except this will be Start-AzureRmWebApp instead of Stop-AzureRmWebApp: powershell -command "Login-AzureRmAccount -Credential (New-Object -TypeName pscredential –ArgumentList 'appId@domain.com', (ConvertTo-SecureString 'password' -AsPlainText –Force)) -ServicePrincipal –TenantId 'tenantId';Start-AzureRmWebApp -ResourceGroupName 'resourcegroup' -Name 'mysite';"

You can also do variations on these commands, such as Stop-AzureRmWebAppSlot and Start-AzureRmWebAppSlot, and use this in combo with Auto Swap to avoid downtime in production while you stop a slot and deploy to the slot: For example, to stop the slot: powershell -command "Login-AzureRmAccount -Credential (New-Object -TypeName pscredential –ArgumentList 'appId@domain.com', (ConvertTo-SecureString 'password' -AsPlainText –Force)) -ServicePrincipal –TenantId 'tenantId';Stop-AzureRmWebAppSlot -ResourceGroupName 'resourcegroup' -Name 'mysite' -slot 'dev';" To disable Always On on a slotless App Service: powershell -command "Login-AzureRmAccount -Credential (New-Object -TypeName pscredential –ArgumentList 'appId@domain.com', (ConvertTo-SecureString 'password' -AsPlainText –Force)) -ServicePrincipal –TenantId 'tenantId';Set-AzureRMResource -PropertyObject @{siteConfig = @{AlwaysOn = $false}} -Name 'mysite' -ResourceGroupName 'resourcegroup' -ResourceType Microsoft.Web/sites -EA SilentlyContinue;" To disable Always On on a slot: powershell -command "Login-AzureRmAccount -Credential (New-Object -TypeName pscredential –ArgumentList 'appId@domain.com', (ConvertTo-SecureString 'password' -AsPlainText –Force)) -ServicePrincipal –TenantId 'tenantId';Set-AzureRMResource -PropertyObject @{siteConfig = @{AlwaysOn = $false}} -Name 'mysite/myslot' -ResourceGroupName 'resourcegroup' -ResourceType Microsoft.Web/sites/slots -EA SilentlyContinue;" Note: replace $false with $true to re-enable Always On. And of course you can combine stopping the site / disabling Always On, as well as starting the site / re-enabling Always On. More information about customizing Kudu deployments can be found at the following links: https://github.com/projectkudu/kudu/wiki/Customizing-deployments https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script Further information about dealing with locked files during deployment can be found at the following link: https://github.com/projectkudu/kudu/wiki/Dealing-with-locked-files-during-deployment

Log in to Azure Resource Manager (ARM) through Kudu to use ARM PowerShell

Gabe Shapiro — Fri, 11 Aug 2017 19:48:53 GMT

Azure App Service has Azure Resource Manage (ARM) PowerShell modules installed, which gives you the ability to run ARM PowerShell commands from the Kudu console and WebJobs (and Azure Functions). In order to use ARM PowerShell within App Service you will need to log in to Azure non-interactively, using a service principal. There are several different ways to create a service principal. I am showing a simple, no-frills way to do so below.

On a local machine, do the following:

1. Interactively log in to ARM PowerShell via the following command: Login-AzureRmAccount 2. Create an Azure Active Directory (AAD)) application: $app = New-AzureRmADApplication -DisplayName "aUniqueAppName" -HomePage "https://myFakeOrRealURI" -IdentifierUris "https://myFakeOrRealURI" -Password "thePasswordOfYourChoice"; 3. Create a Service Principal for the app: New-AzureRmADServicePrincipal -ApplicationId $app.ApplicationId; 4. Assign the Service Principal a role on the app: New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $app.ApplicationId; 5. In preparation for the next set of steps, get the following values: -Tenant Id: (Get-AzureRmSubscription).TenantId; -ApplicationId: $app.ApplicationId; -Tenant domain name: This would be the domain name in the Account property, which shows when you log in via login-azurermaccount with your user account.

Now, in Kudu, you can run the following script from the PowerShell console to log into Azure ARM, and then you can run other ARM commands:

$tenantId = "yourTenantId"; $domain = "yourDomain"; $appId = "yourApplicationId"; $username = $appId + '@' + $domain; $pass = ConvertTo-SecureString "thePasswordOfYourChoice" -AsPlainText –Force; $cred = New-Object -TypeName pscredential –ArgumentList $username, $pass; Login-AzureRmAccount -Credential $cred -ServicePrincipal –TenantId $tenantId;

Using the Azure ARM SDK for Node to get Site Metrics for your App Service

Gabe Shapiro — Sun, 01 Jan 2017 23:16:18 GMT

Documentation for using the Azure ARM SDK for Node to get Site Metrics for your App Service can be found here: https://azure.github.io/azure-sdk-for-node/azure-arm-website/latest/Sites.html#getSiteMetrics Here is sample code for how to get this to return the results as JSON in the console. You can use other login methods (not just loginWithServicePrincipalSecret), such as interactiveLogin.

var msRestAzure = require('ms-rest-azure');

var webSiteManagementClient = require('azure-arm-website');

msRestAzure.loginWithServicePrincipalSecret(clientId, secret, domain, function(err, credentials) { if (err) { console.log(err); return; } var client = new webSiteManagementClient(credentials, subscription);

var options = { filter: 'name.value eq \'Requests\' and startTime eq 2016-12-017T19:40:00Z and endTime eq 2016-12-17T19:55:00Z and timeGrain eq duration\'PT1M\'', customHeaders: {accept: 'application/json'} };

try { result = client.sites.getSiteMetrics(resourceGroup, name, options, function(err, result) { if (err) console.log(err); console.log("RESULT = " + JSON.stringify(result, null, 2)); }); } finally { } });

Note: Based on this documentation, the retention policy for each granularity is currently as follows: -Minute granularity metrics (eg PT1M) are retained for 48 hours -Hour granularity metrics (eg PT1H) are retained for 30 days -Day granularity metrics (eg PT1D) are retained for 90 days

You can use the Microsoft Azure Cross Platform Command-Line (among other tools) to run the Node application. More information about the Azure Cross Platform Command-Line can be found here:

https://github.com/Azure/azure-xplat-cli

Samples for using the Azure App Service Kudu REST API to programmatically manage files in your site

Gabe Shapiro — Sun, 01 Jan 2017 22:54:49 GMT

Information about the Kudu REST API is found here: https://github.com/projectkudu/kudu/wiki/REST-API The VFS API section contains examples for programmatically managing files and directories in the App Service. Here is an example (written in PowerShell) for listing the files and folders in the wwwroot folder. $username = "`$websitename"; $password = "password"; <# This is the password from the msdeploySite credentials in the publish profile. You can get the publish profile by going to the Overview blade on your App Service, clicking ...More at the top of the blade, and then clicking Get publish profile. #> $base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password))) $userAgent = "powershell/1.0"; $apiUrl = "https://websitename.scm.azurewebsites.net/api/vfs/site/wwwroot/"; Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method GET -ContentType "application/json"; Here is an example that puts a local file from your machine into the App Service file system: $filePath = "c:\temp\text.txt"; $apiUrl = "https://websitename.scm.azurewebsites.net/api/vfs/site/wwwroot/test.txt"; Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method PUT -InFile $filePath -ContentType "multipart/form-data"; Here is an example that gets a file from the App Service file system and puts it in an existing folder on your local machine: $filePath = "c:\temp\text.txt"; $apiUrl = "https://websitename.scm.azurewebsites.net/api/vfs/site/wwwroot/test.txt"; Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method GET -OutFile $filePath -ContentType "multipart/form-data"; Here is an example that gets a folder on the App Service file system (the wwwroot folder in this case) and puts its contents on your local machine as a zip file: $filePath = "c:\temp\wwwroot.zip"; $apiUrl = "https://websitename.scm.azurewebsites.net/api/zip/site/wwwroot/"; Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -UserAgent $userAgent -Method GET -OutFile $filePath -ContentType "application/zip";

Umbraco logging in Azure App Service

Gabe Shapiro — Sun, 01 Jan 2017 22:26:47 GMT

Umbraco is a third-party CMS that can be hosted in Azure App Service. Umbraco has built-in Log4Net logging that can be useful for troubleshooting issues within an Umbraco application. In Azure App Service, these logs can typically be found under D:\home\site\wwwroot\App_Data\Logs. The naming convention for these logs is UmbracoTraceLog.InstanceName.txt . Note: InstanceName refers to a machine instance that hosts your App Service, so if your App Service Plan has multiple instances or if the apps are moved to a new instance over time, you may see multiple log files in this folder. You can view this logging configuration under D:\home\site\wwwroot\Config\log4net.config , and can change the logging level (eg DEBUG, INFO, WARN, ERROR, or FATAL). The default logging level is WARN. You can access these files using the App Service's Kudu console at https://.scm.azurewebsites.net/DebugConsole or by using an FTP client. For example, I was troubleshooting an issue where the application pool was recycling when files were published to Umbraco. In the Umbraco logs, I found the following event, which indicated the reason for this behavior: yyyy-mm-dd hh:mm:ss [P<processId>/D<appDomainId>/T<thread>] INFO Umbraco.Core.UmbracoApplicationBase - Application shutdown. Details: HostingEnvironment _shutDownMessage=Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources HostingEnvironment initiated shutdown CONFIG change I learned that this behavior was related to the fcnMode setting that the application was using. Umbraco uses fcnMode=Single by default, which is beneficial in many scenarios in that it automatically restarts the app domain when certain app changes are made, so that you don't need to do a site restart. However, in certain cases, the buffer can become overwhelmed since Single mode uses a single buffer for all subdirectories, and can cause the application pool to recycle (site restart). A workaround is to use fcnMode=Disabled, which means that you would need to restart the site in order for the app changes to show up. The potential benefit to this approach is that you have more control over when the application pool recycles. More information about the fcnMode behavior in Umbraco can be found here. The fcnMode setting, if present in the web.config, would be found under System.Web, in the httpRuntime element. For example: