SQL Server Premier Field Engineer Blog

Physical_Device_Name Not Updated On Restore

Ryan J. Adams — Mon, 25 Feb 2019 21:27:53 GMT

The Problem

You restored a database on top of an existing database to refresh your development environment, but the Physical_Device_Name in msdb..backupmediafamily does not get updated. That seems odd, but let's take a look at the documentation on that table. We can find the documentation here on Backup Media Family. Zooming in on the Physical_Device_Name column we see this (Click for a larger image).

Verify

Trust but verify! Let's try to reproduce this and see if the documentation is correct. Here are the steps we'll take.

Step 1
- Put a backup file in C:\Temp, restore it, and then check the physical_device_name column in msdb..backupmediafamily
Result 1
- We see C:\Temp\MyBackup.bak
Step 2
- Move the backup file to C:\Temp\Test, restore it again, and check the physical_device_name column in msdb..backupmediafamily.
Result 2
- We see C:\Temp\MyBackup.bak

Well that's not at all what we expected was it? We expected to see C:\Temp\Test\MyBackup.bak the second time we checked. Well that verifies the documentation. Here is the query we used to verify. [code language="sql"] SELECT [rs].[destination_database_name] , [rs].[restore_date] , [bs].[backup_start_date] , [bs].[backup_finish_date] , [bs].[database_name] AS [source_database_name] , [bmf].[physical_device_name] FROM msdb..restorehistory rs INNER JOIN msdb..backupset bs ON [rs].[backup_set_id] = [bs].[backup_set_id] INNER JOIN msdb..backupmediafamily bmf ON [bs].[media_set_id] = [bmf].[media_set_id] ORDER BY [rs].[restore_date] DESC [/code]

Solutions

Let's take a look at some options to get that column updated with the most recent and correct information. We are also going to include options for just finding the correct data in other places.

Drop the database prior to the restore (This deletes the backup/restore history as well)
Delete the backup/restore history prior to the restore using sp_delete_backuphistory
Create an XE session for database_recovery_progress_report events and include the sql_text field. I would also filter on phase so we don’t waste space capturing all of the restore steps and only see the completion.
Query the error log

I'm not going to cover the first two options since they are quite simplistic, but let's take a look at how we could use the last two options.

Extended Event

This does not fix what gets stored in the msdb..backupmediafamily table, but is an alternate way to track and store restore information with the accurate information we desire. We can track this information using the sqlserver.database_recovery_progress_report event in Extended Events. Here is what that would look like: [code language="sql"] CREATE EVENT SESSION [Restore Tracking] ON SERVER ADD EVENT sqlserver.database_recovery_progress_report(SET collect_database_name=(1) ACTION(sqlserver.sql_text) WHERE ([phase]=(4))) ADD TARGET package0.event_file(SET filename=N'Restore Tracking') WITH (STARTUP_STATE=ON) GO ALTER EVENT SESSION [Restore Tracking] ON SERVER STATE = start; GO [/code] You can read more about Extended Events for Database Recovery HERE.

Error Log

Even though the Physical_Device_Name does not get updated it is logged correctly in the SQL Server error log. Here is what that would look like. Note that parameter 5 and 6 allow for start and finish times if you need to narrow that down, but I left them NULL here to search the whole log. [code language="sql"] EXEC master.dbo.xp_readerrorlog 0, 1, N'Database was restored', NULL, NULL, NULL, N'asc' [/code]

Automate Azure VM Storage

Ryan J. Adams — Mon, 25 Feb 2019 21:18:44 GMT

The Problem

One of the best things about Azure, and the cloud in general, is we can automate most anything, and we are going to look at how to automate Azure VM Storage. This allows us to come up with some outside-of-the-box solutions. I had a customer with a road block that we were able to work around by automating some things with their Azure Virtual Machines. Their challenge was that they wanted to move their test and development environments to Azure, but the storage cost was prohibitive. They needed premium storage to mimic their production environment, but it was not financially viable for test and development so they were going to keep it all on premises. During our conversations I learned that they only test between 8am and 5pm, Monday through Friday. My suggestion was that we put their databases on cheaper storage during off times and only premium when they are actively using it.

The Plan

Here is where the automation comes into play. My challenge was to see if I could find a way to automate moving everything between standard and premium storage. Here is my plan:

Create a Storage Account
- Create a container for my backups
- Create a container for my Custom Script Extensions and SQL scripts
- Create a credential for SQL to access my Storage Account
Create an Azure Automation Account
- Create Runbooks
Add Premium Storage Drive
- Create a Premium Azure Disk in my Storage Account
- Add the Premium disk to my VM
- Initialize and format the new disk in the VM
Move the Database
- Restore the database from blob storage to the new Premium Drive
Remove Premium Storage Drive
- Backup database to blob storage
- Restore database to the standard disk
- Remove the Premium disk from the OS
- Remove the Premium disk blob from Azure

This looks like a good plan if I do say so myself. I'll jump out on a limb and place a bet that it works!

Storage Account for Backups

We need to get some things prepared before we jump into any automation. The first thing we need is a storage account where we can store backups of our database. As we move between standard and premium storage, we'll use this to both backup to and restore from. We can use an existing account or create a new one. Once we have the account we need to create a container to house the backups. The other thing we have to think about is creating a credential in SQL to access this container. We only need to do this once so let's go ahead and do that now. Here is what our code looks like so far: [code language="powershell"] #install-module AzureRM import-module AzureRM # Define global variables for the script. Make sure to change these to the names you want to use on your account. $prefixName = 'storagetest' # used as the prefix for the name for various objects $subscriptionID='MySubID' # the name of subscription name you will use $locationName = 'South Central US' # the data center region you will use $storageAccountName= $prefixName + 'allbackups' # the storage account name you will create or use $containerName= $prefixName + 'server1backups' # the storage container name to which you will attach the SAS policy with its SAS token $policyName = $prefixName + 'sqlbackups' # the name of the SAS policy $resourceGroupName= 'VMStorageTestRM' # Name of the resource group you will create or use # adds an authenticated Azure account for use in the session Login-AzureRmAccount # set the tenant, subscription and environment for use in the rest of Set-AzureRmContext -SubscriptionID $subscriptionID # create a new resource group - comment out this line to use an existing resource group New-AzureRmResourceGroup -Name $resourceGroupName -Location $locationName # Create a new storage account - comment out this line to use an existing storage account New-AzureRmStorageAccount -Name $storageAccountName -ResourceGroupName $resourceGroupName -Type Standard_RAGRS -Location $locationName # Get the access keys for the ARM storage account $accountKeys = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -AccountName $storageAccountName).Value[0] # Create a new storage account context using a storage account $storageContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $accountKeys # Creates a new container in blob storage $container = New-AzureStorageContainer -Context $storageContext -Name $containerName $cbc = $container.CloudBlobContainer # Sets up a Stored Access Policy and a Shared Access Signature for the new container. Note that it expires in 10 years. $permissions = $cbc.GetPermissions(); $policy = new-object 'Microsoft.WindowsAzure.Storage.Blob.SharedAccessBlobPolicy' $policy.SharedAccessStartTime = $(Get-Date).ToUniversalTime().AddMinutes(-5) $policy.SharedAccessExpiryTime = $(Get-Date).ToUniversalTime().AddYears(10) $policy.Permissions = "Read,Write,List,Delete" $permissions.SharedAccessPolicies.Add($policyName, $policy) $cbc.SetPermissions($permissions); # Get the Shared Access Signature for the policy $policy = new-object 'Microsoft.WindowsAzure.Storage.Blob.SharedAccessBlobPolicy' $sas = $cbc.GetSharedAccessSignature($policy, $policyName) Write-Host 'Shared Access Signature= '$($sas.Substring(1))'' # Outputs the Transact SQL to the clipboard and to the screen to create the credential using the Shared Access Signature Write-Host 'Credential T-SQL' $tSql = "CREATE CREDENTIAL [{0}] WITH IDENTITY='SHARED ACCESS SIGNATURE', SECRET='{1}'" -f $cbc.Uri,$sas.Substring(1) $tSql | clip Write-Host $tSql [/code] Once the script completes, you will get 2 pieces of output. The first is the Shared Access Signature. The second is the TSQL you will need to create a credential in SQL Server to access this container. You do not even have to copy and paste it. The code automatically puts it in your clip board, you just need to go paste it into SSMS connected to your SQL Server. Here is an example of what it will look like: [code language="sql"] CREATE CREDENTIAL [https://MyStorageAccountName.blob.core.windows.net/MyStorageContainerName] WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = 'YourSecretWillBeHere'; [/code]

Add Premium Disk

The next step is to add a premium disk to our VM. This piece of code is going to be placed into an Azure Automation Account Runbook. I thought this part would be really easy and it is easy to add the disk to the VM in Azure, but what gets tricky is formatting it within the VM. I don't want to login to the VM, initialize the disk, and format it. We are automating this for a reason after all. The good news is we have something called Custom Script Extensions that will take a PowerShell script sitting in a blob container and execute it locally on the VM. It took some time to finesse this part out, but it's a really cool power to have, and it allows us to do pretty much anything we want. I'm not going to walk through that, but you can go here to read more about Azure Custom Script Extensions. In the code for our new Runbook you will see that I use the Custom Script Extension to call, "4_Initialize_Format_VM_Disk.ps1". That script does exactly what it says in the name and we'll look at that in the next section. Here is the code to create a Premium disk in Azure and add it to our SQL VM. [code language="powershell"] # Define global variables for the script $subscriptionID='MySubID' # the ID of subscription name you will use $resourceGroupName= 'VMStorageTestRM' $myVMName = "VMStorageTest" #The name of our Azure VM running SQL Server $locationName = 'South Central US' $fileName = '4_Initialize_Format_VM_Disk.ps1' #Used with our Custom Script Extension to initialize and format the drive within the VM $scriptName = '4_Initialize_Format_VM_Disk.ps1' #Used with our Custom Script Extension to initialize and format the drive within the VM $storageAccountName= 'storagetestallbackups' #This is the same storage account we created in the previous step $typeHandlerVersion = '1.8' $storageContainer = 'vmscripts' #This is a different container in our storage account to hold our Custom Script Extensions $connectionName = "AzureRunAsConnection" try { # Get the connection "AzureRunAsConnection " $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName "Logging in to Azure..." Add-AzureRmAccount ` -ServicePrincipal ` -TenantId $servicePrincipalConnection.TenantId ` -ApplicationId $servicePrincipalConnection.ApplicationId ` -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint } catch { if (!$servicePrincipalConnection) { $ErrorMessage = "Connection $connectionName not found." throw $ErrorMessage } else{ Write-Error -Message $_.Exception throw $_.Exception } } # Add the premium drive to the VM $VirtualMachine = Get-AzureRmVM -ResourceGroupName $ResourceGroupName -Name $myVMName Add-AzureRmVMDataDisk -VM $VirtualMachine -Name "premiumdisk1" -LUN 2 -Caching ReadWrite -DiskSizeinGB 1 -CreateOption Empty -StorageAccountType PremiumLRS Update-AzureRmVM -ResourceGroupName $ResourceGroupName -VM $VirtualMachine # Get the access keys for the storage account $accountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -AccountName $storageAccountName).Value[0] # Set and execute the custom script extension. Initializes and formats drive then backs up restores DB to new drive Set-AzureRmVMCustomScriptExtension -ResourceGroupName $resourceGroupName -Location $locationName -VMName $myVMName -Name $scriptName -Run $scriptName ` -TypeHandlerVersion $typeHandlerVersion -StorageAccountName $StorageAccountName -StorageAccountKey $accountKey -FileName $fileName -ContainerName $storageContainer Update-AzureRmVM -ResourceGroupName $resourceGroupName -VM $VirtualMachine #Good to keep it clean and remove our extension. The extension also has a password so we really want to cleanup here. Remove-AzureRmVMCustomScriptExtension -ResourceGroupName $resourceGroupName -Name $scriptName -VMName $myVMName -Force [/code]

Initialize and Format

It's time to write that Custom Script Extension that will execute locally on our SQL VM to initialize and format our newly added Premium Disk. First we need to grab a reference to our new drive (formatting the C:\ drive is still considered really bad form). Since the disk is brand new we can identify it by its partition type being "raw". After that we grab a letter from the alphabet, initialize it, format it, and assign the letter. We also create folders on the new drive to house our data and log files. This script is also going to set us up for the next step of restoring our database out of blob storage and onto the new drive. For this we will create a new folder on the VM to house a SQL script that will do the restore for us. Next we will download that script from our storage account, and finish by executing it. We are going to call the SQL script "5_Azure_Backup_Restore_To_Premium.sql" and we will discuss that in the following section. For now, let's look at our code to initialize our new VM disk, format it, and set the stage for the next script. [code language="powershell"] # Setup variables $storageAccountName = "storagetestallbackups" $resourceGroupName = "VMStorageTestRM" $subscriptionId = "YourAzureSubscriptionID" # Now that we added the disk in Azure we need to initialize and format it in the OS $disks = Get-Disk | Where partitionstyle -eq 'raw' | sort number $letters = 71..89 | ForEach-Object { [char]$_ } $count = 0 $labelprefix = "data" foreach ($disk in $disks) { $driveLetter = $letters[$count].ToString() $disk | Initialize-Disk -PartitionStyle MBR -PassThru | New-Partition -UseMaximumSize -DriveLetter $driveLetter | Format-Volume -FileSystem NTFS -NewFileSystemLabel ($labelprefix + $count) -AllocationUnitSize 65536 -Confirm:$false -Force $count++ } # We need to add the directories we plan to restore to before we do the restore New-Item G:\Data -type directory New-Item G:\Log -type directory # Create a directory to store our local SQL script New-Item C:\Scripts -type directory -Force ############ This script is executing locally on the VM as a Custom Script Extension so we need to authenticate to Azure ################## $pwd = "%SupersweetPassword%" #Don't panic. This script is stored in a secured storage container and removed from the VM after it runs. #$azureAdApplication.ApplicationId.Guid #If you are running this in the same session as the setup this will give you the application ID in case you forgot it $accountId = "YourAccountID" $tenantId = "YourTenantID" #You get this from just running Login-AzureRmAccount and logging in at the prompt. $login = $AccountId.ToString() + "@microsoft.onmicrosoft.com" #Create Credentials $pass = ConvertTo-SecureString $pwd -AsPlainText –Force $cred = New-Object -TypeName pscredential –ArgumentList $login, $pass #Login Login-AzureRmAccount -Credential $cred -ServicePrincipal –TenantId $tenantId ############################################################################################### # Set context to the desired subscription Set-AzureRmContext -SubscriptionID $subscriptionID # Get the access keys for the storage account $accountKeys = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -AccountName $storageAccountName).Value[0] # Create a new storage account context using an ARM storage account $storageContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $accountKeys # Download the script from Azure to our new folder $storageContext | Get-AzureStorageBlobContent -Container vmscripts -Blob 5_Azure_Backup_Restore_To_Premium.sql -Destination 'C:\Scripts' -Force # Run our SQL script Invoke-Sqlcmd -InputFile 'C:\Scripts\5_Azure_Backup_Restore_To_Premium.sql' -Username 'AzureAutomation' -Password 'AA123' #You will need to use a SQL account in your target instance [/code]

Move the Database

The previous script kicked off "5_Azure_Backup_Restore_To_Premium.sql" at the very end, which handles moving our database. Remember that the database currently exists and is running on the standard storage drive. I chose to back it up using the "TO URL" option of the "BACKUP DATABASE" command and backup the database to Azure blob storage. Next I restore it from Azure blob storage to my new drive. This is not the only way I could have done this, but I like the consistency and the fact that I am backing up my database every time I make a major change. I could have just as easily backed it up to a local drive and restored from there. I even could have detached it and attached it, but it is less risky to have a backup using checksum. Let's see what this script looks like. [code language="sql"] /* Put the DB in SIMPLE Recovery */ ALTER DATABASE [Example] SET RECOVERY SIMPLE GO /* Perform a Full Backup */ BACKUP DATABASE [Example] TO URL = 'https://storagetestallbackups.blob.core.windows.net/storagetestserver1backups/Example_Full_1.bak' WITH FORMAT, INIT, COMPRESSION, CHECKSUM GO /* Restore the full backup to the new drive */ USE [master] GO ALTER DATABASE [Example] SET SINGLE_USER WITH ROLLBACK IMMEDIATE RESTORE DATABASE [Example] FROM URL = 'https://storagetestallbackups.blob.core.windows.net/storagetestserver1backups/Example_Full_1.bak' WITH MOVE 'Example' TO 'G:\Data\Example.mdf', MOVE 'Example_Log' TO 'G:\Log\Example_Log.ldf', GO ALTER DATABASE [Example] SET MULTI_USER GO [/code]

Reverse It

Things are looking really good now. We have successfully created a Runbook that adds a Premium storage disk to our VM and moves our database to the new drive. That's great because we can manually kick it off in the morning or schedule it to run automatically so we are ready to do our testing or development work for the day. The last thing we need to do is reverse all of this and put everything back on a standard storage disk. Let's work on that. We have already created our storage account and created a credential to access it, so we don't have to do all that work again. We just need to move the database back to the standard storage drive and get rid of the premium drive. We'll start by creating our second Runbook to handle this.

Remove the Premium Disk

This Runbook is again a chain of scripts to accomplish what we want to achieve. The code will make use of our Custom Script Extension to fire off another script that will execute locally on the VM. The local script will download a SQL script to move our database. Once that is done our Runbook code will remove the Premium drive from our VM. Here is what the Runbook code looks like: [code language="powershell"] # Define global variables for the script $resourceGroupName= 'VMStorageTestRM' $myVMName = "VMStorageTest" $storageAccountName = 'storagetestallbackups' $locationName = 'South Central US' $scriptName = '4-1_Execute_Restore_To_Standard.ps1' $fileName = '4-1_Execute_Restore_To_Standard.ps1' $typeHandlerVersion = '1.8' $storageContainer = 'vmscripts' #Connect to Azure $connectionName = "AzureRunAsConnection" try { # Get the connection "AzureRunAsConnection " $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName "Logging in to Azure..." Add-AzureRmAccount ` -ServicePrincipal ` -TenantId $servicePrincipalConnection.TenantId ` -ApplicationId $servicePrincipalConnection.ApplicationId ` -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint } catch { if (!$servicePrincipalConnection) { $ErrorMessage = "Connection $connectionName not found." throw $ErrorMessage } else{ Write-Error -Message $_.Exception throw $_.Exception } } # Get the access keys for the storage account $accountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -AccountName $storageAccountName).Value[0] # Grab VM object reference $VirtualMachine = Get-AzureRmVM -ResourceGroupName $ResourceGroupName -Name $myVMName # Set the custom extension Set-AzureRmVMCustomScriptExtension -ResourceGroupName $resourceGroupName -Location $locationName -VMName $myVMName -Name $scriptName -Run $scriptName ` -TypeHandlerVersion $typeHandlerVersion -StorageAccountName $StorageAccountName -StorageAccountKey $accountKey -FileName $fileName -ContainerName $storageContainer # Update the VM. This executes the custom script extension on the VM. The extension uses a SQL script to move the DB to standard storage Update-AzureRmVM -ResourceGroupName $resourceGroupName -VM $VirtualMachine #We want to remove our script extension from the VM here Remove-AzureRmVMCustomScriptExtension -ResourceGroupName $resourceGroupName -VMName $myVMName -Name $scriptName -Force Update-AzureRmVM -ResourceGroupName $resourceGroupName -VM $VirtualMachine #This will detach the disk from the VM and the OS will automatically remove it Remove-AzureRmVMDataDisk -VM $VirtualMachine -Name "premiumdisk1" Update-AzureRmVM -ResourceGroupName $ResourceGroupName -VM $VirtualMachine #We detached it above, but that doesn't remove the VHD blob from Azure. Remove-AzureRmDisk -DiskName "premiumdisk1" -ResourceGroupName $resourceGroupName -Force [/code]

Move the Database

The code above in our Runbook calls a Custom Script Extension so let's inspect it and see what it does. [code language="powershell"] # Setup variables $storageAccountName = "storagetestallbackups" $resourceGroupName = "VMStorageTestRM" $subscriptionId = "YourSubscriptionID" # Create a directory to store our local SQL script New-Item C:\Scripts -type directory -Force ############ Use this section in other scripts to authenticate going forward ################## $pwd = "%SupersweetPassword%" #$azureAdApplication.ApplicationId.Guid #If you are running this in the same session as the setup this will give you the application ID in case you forgot it $accountId = "YourAccountID" $tenantId = "YourTenantID" #You get this from just running Login-AzureRmAccount and logging in at the prompt. $login = $AccountId.ToString() + "@microsoft.onmicrosoft.com" #Create Credentials $pass = ConvertTo-SecureString $pwd -AsPlainText –Force $cred = New-Object -TypeName pscredential –ArgumentList $login, $pass #Login Login-AzureRmAccount -Credential $cred -ServicePrincipal –TenantId $tenantId ############################################################################################### # Set context to the desired subscription Set-AzureRmContext -SubscriptionID $subscriptionID # Get the access keys for the ARM storage account $accountKeys = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -AccountName $storageAccountName).Value[0] # Create a new storage account context using an ARM storage account $storageContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $accountKeys # Download the script from Azure to our new folder $storageContext | Get-AzureStorageBlobContent -Container vmscripts -Blob 5-1_Azure_Backup_Restore_To_Standard.sql -Destination 'C:\Scripts' -Force # This runs a SQL script to backed and restore the DB to the Standard storage disk Invoke-Sqlcmd -InputFile 'C:\Scripts\5-1_Azure_Backup_Restore_To_Standard.sql' -Username 'AzureAutomation' -Password 'AA123' [/code] The code here is pretty simple compared to what we wrote to add the Premium drive. All we do is download a SQL script from our storage account and execute it on the VM. The SQL script will move the database to standard storage. It should be pretty obvious what the SQL script looks like, but let's take a peak.

Restore to Standard

Nothing surprising here. We backup our Example database to our blob storage account and then restore it using the WITH MOVE operation to put it back on our standard storage drive. Totally straightforward, so here it is: [code language="sql"] /* Put the DB in SIMPLE Recovery */ ALTER DATABASE [Example] SET RECOVERY SIMPLE GO /* Perform a Full Backup */ BACKUP DATABASE [Example] TO URL = 'https://storagetestallbackups.blob.core.windows.net/storagetestserver1backups/Example_Full_1.bak' WITH FORMAT, INIT, COMPRESSION, CHECKSUM GO /* Restore the full backup */ USE [master] GO ALTER DATABASE [Example] SET SINGLE_USER WITH ROLLBACK IMMEDIATE RESTORE DATABASE [Example] FROM URL = 'https://storagetestallbackups.blob.core.windows.net/storagetestserver1backups/Example_Full_1.bak' WITH MOVE 'Example' TO 'F:\Data\Example.mdf', MOVE 'Example_Log' TO 'F:\Log\Example_Log.ldf', GO ALTER DATABASE [Example] SET MULTI_USER GO [/code]

Food for Thought

Here is a list of things that could be altered to streamline this process if you were so inclined.

The code to create the credential for SQL to access your storage accont could just be called through the Custom Script Extension instead of the copy and paste method we used here.
When I initialize the disk in the VM I know what drive letter I want because I know what my VM looks like. It would be better to check the highest currently used drive letter and grab the next one. This was a contrived example so I knew I wanted G:\.
Instead of using separate scripts for the SQL code, this code could be brought into the Custom Script Extension. This would allow for the above change of dynamically choosing the drive letter.
We backed up the database to blob storage and then restored it. We could have restored it to a local drive (the standard storage drive most likely) and restored from there. This would be faster with a larger database. Mine was really small so it was no big deal and I like the idea of backups right before a major change plus getting them off of the box I am working on. If the box fails then my backups are no good.

Conclusion

This customer wanted to move to Azure for their test and development systems, but they wanted to do it in the most cost efficient way. Only paying for what you use is one of the bigger value propositions of the cloud. Sometimes we have to think outside the box to accomplish our goal and that is exactly what we just accomplished in this post. It might seem like a lot of work, but I just gave you a giant leap in the right direction. Enjoy automating your Azure VMs!

FileStream Backup and Restore

DayoBam — Fri, 05 Oct 2018 19:48:53 GMT

I have been asked to provide some information on best practice for SQL Server Backup and Restore with Filestream Databases. For Backup. The method is exactly the same like you do for your regular non-Filestream databases. This means you can include your Filestream database in your current backup strategy with no additional work. For Restore. There is a bit of understanding that is required. Filestream has the following to restore

 Data

 Log

 Filestream Data

The last part Filestream Data is what you need to be aware of. For example if I want to restore a Filestream database unto the same instance, I can't just rename the database, because that only changes the data and log name, the Filestream Data which is stored in a directory on the same environment needs to be modified as well. The image below is where you can do this. I recommend using a folder completely different to avoid any issues. If you have more question around this. Please let me know If you're restoring to a new environment. The Filestream data and log will be restore to the default data directory and so will the Filestream data. If you do not want it in the Data directory, you will have to specify that like in the image below. Here is a Microsoft blog that explains the restore process for Filestream How to: Restoring SQL File stream Data https://blogs.msdn.microsoft.com/sql_pfe_blog/2013/07/23/how-to-restoring-sql-file-stream-data/ This is another resource that I found that might be useful. The book has a lot of information about Filestream https://assets.red-gate.com/community/books/art-of-ss-filestream.pdf Please let me know if you have any other questions. ------------ One thing I forgot to mention is that when you restore the filestream database on the same instance Make sure you are aware of the tail-log backup. You should have this unchecked. If you don't, you will but the original database in a restoring state, meaning no one can access the database, until it is recovered. If you have the tail-log check here is what the script looks like USE [master] BACKUP LOG [Archive] TO DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Backup\Archive_LogBackup_2018-08-16_11-36-52.bak' WITH NOFORMAT, NOINIT, NAME = N'Archive_LogBackup_2018-08-16_11-36-52', NOSKIP, NOREWIND, NOUNLOAD, NORECOVERY , STATS = 5 RESTORE DATABASE [1234] FROM DISK = N'C:\Archive.bak' WITH FILE = 1, MOVE N'Arch1' TO N'C:\FSDemo\archdat1.mdf', MOVE N'Archlog1' TO N'C:\FSDemo\archlog1.ldf', MOVE N'Arch3' TO N'C:\FSDemo\filestream1', MOVE N'arch4' TO N'C:\FSDemo\filestream2', MOVE N'arch5' TO N'C:\FSDemo\filestream3', NOUNLOAD, STATS = 5 GO Notice the second line is taking a backup of the original database, when this is done, this puts the original database in a restore mode. If you accidently do this. You have to recover the database with the following script RESTORE DATABASE WITH NO RECOVERY Ex. RESTORE DATABASE Archive WITH NO RECOVERY This should recover the database and business can continue as usual. Here is a script without the tail-log. This will work fine without affecting the original database. USE [master] RESTORE DATABASE [1234] FROM DISK = N'C:\Archive.bak' WITH FILE = 1, MOVE N'Arch1' TO N'C:\FSDemo\archdat1.mdf', MOVE N'Archlog1' TO N'C:\FSDemo\archlog1.ldf', MOVE N'Arch3' TO N'C:\FSDemo\filestream1', MOVE N'arch4' TO N'C:\FSDemo\filestream2', MOVE N'arch5' TO N'C:\FSDemo\filestream3', NOUNLOAD, STATS = 5 GO

AlwaysOn Availability Groups unable to query against readable secondary replica database: Wait Type HADR_DATABASE_WAIT_FOR_TRANSITION_TO_VERSIONING

Chirag Shah — Fri, 13 Jul 2018 09:43:41 GMT

Recently a customer reported an interesting issue, while querying against recently added readable replica, SELECT statement is shown as suspended and session is shown as waiting on HADR_DATABASE_WAIT_FOR_TRANSITION_TO_VERSIONING Here is a simple way to reproduce the scenario Now on current primary replica, change one of the remote replica to readable However, running a SELECT Statement against newly available readable replica, query appeared to be "suspended" Upon more investigation, it appeared to be waiting on with a wait type HADR_DATABASE_WAIT_FOR_TRANSITION_TO_VERSIONING The behavior is by design as mention in the SQL Server product documentation and applicable to all version of SQL Server that supports availability group. "As soon as a readable secondary replica joins the availability group, the secondary replica can start accepting connections to its secondary databases. However, if any active transactions exist on a primary database, row versions will not be fully available immediately on the corresponding secondary database. Any active transactions that existed on the primary replica when the secondary replica was configured must be committed or rolled back. Until this process completes, the transaction isolation level mapping on the secondary database is incomplete and queries are temporarily blocked." This signals that there is at least one transaction open on the primary system which had been opened before you switched the secondary to become a readable secondary. As mentioned, when switching to a readable secondary, we need to provide committed versions before the Redo Thread is performing changes to the data. However, with a transaction still open on the primary that has been open before switching to readable secondary, such a committed version of the data could not be provided on the secondary. Hence the select will wait until the transition is successful. The transition becomes successful, when all the transactions that were open before switching are committed or rolled back on the primary.

Configure SQL Server Failover Cluster Instance on Azure Virtual Machines with MSDTC

Ryan J. Adams — Thu, 05 Jul 2018 08:00:59 GMT

Configure SQL Server Failover Cluster Instance

If you are running a SQL Server Failover Cluster Instance on premises and looking to migrate to Azure, you might be a little intimidated. This architecture is a bit complex because it uses quite a few different technologies all working together. My goal is to help clarify the pieces and steps needed to build this out. There is already a great article that walks through the configuration. It has links to supplemental information as well, so it's an excellent reference guide. There is, however, one thing it does not cover. It does not show how to configure a clustered MSDTC. In this article I will list the pieces required to make this work and give you a video walk through on the entire configuration. Read this article first to become familiar with the steps.

Technologies Involved

Windows Server
Windows Clustering
Storage Spaces Direct (S2D)
Cluster Shared Volumes
SQL Server
MSDTC

Configuration Steps

Create Virtual Machines
- Must be in an Availability Set
- Use Standard IPs for MSDTC
- Have at least 2 data disks
Add Windows Failover Cluster Feature
Create Windows Failover Cluster
- Create a witness if needed
Make sure data disk have no partitions
Enable Storage Spaces Direct (S2D)
Create volumes from S2D pool
Install SQL Server Failover Cluster Instance
Install a clustered MSDTC
Create an Internal Azure Load Balancer
- Needs to be a Standard Load Balancer for MSDTC (VM IPs and Load Balancers must match SKUs)
- Front end for SQL and one for the MSDTC
- Health probe for SQL and the MSDTC
- Load balancing rule for SQL and the MSDTC
Configure SQL IP for probe port
Configure MSDTC IP for probe port

MSDTC Requirements

To make a clustered MSDTC work in this architecture we need to make sure of a couple things. We need to use a Standard Load Balancer instead of a Basic Load Balancer. Since we have to use a Standard Load Balancer that means the IPs for our VMs also have to be Standard IPs. You can only add Basic IPs to a Basic Load Balancer and Standard IPs to a Standard Load Balancer. The other requirement is the Windows version. The MSDTC did not support using a Cluster Shared Volume as its shared storage when Windows Server 2016 was released. Support for that was not introduced until Windows Server 2016 Release Build Version 1709. Although the support was added to release build versions 1709 and 1803, neither of those builds include the desktop experience (No GUI). In Windows Server 2019 we get both the Desktop Experience as well as support for the MSDTC to use Cluster Shared Volumes.

Configuration Walk Through

Here is a video where I walk through the entire build process of this architecture including both SQL and the MSDTC. Note that this is just for demonstration purposes and not following all best practices for building Windows Server Failover Clusters. For instance, I do not even configure a witness which you would absolutely do for a production build. https://youtu.be/GS12sfOdC1o

Resources

MSDTC support and configuration can be a bit confusing. Read this article on MSDTC Supported Configurations for more information.

Domain Independent Cluster and Availability Group

Ryan J. Adams — Thu, 05 Jul 2018 08:00:53 GMT

What is a Domain Independent Cluster?

Did you know that you can have a Windows cluster that is a domain independent cluster and run a SQL Server Availability Group on top of it? The technology has been around for a while, but a lot of DBAs are not aware of it. Why? All the glory belongs to Windows since that is where the magic resides. It's like SQL is an awesome Ninja Cat riding on a magic unicorn! [caption id="" align="alignnone" width="168"] Come at me Bro![/caption]

Requirements

Okay, now that you have had time to calm down from all the awesomeness, let's talk about the requirements. First we need Windows and the requirement is Windows Server 2016 or above. The keen observer might have caught that "or above" I slipped into that sentence. If you are not part of the Windows Insider program, you should check it out. There is some really cool stuff coming in Windows that is directly applicable to clusters. If you build or support SQL Server AlwaysOn Failover Cluster Instances or SQL Server AlwaysOn Availability Groups, you should be keeping up with Windows Server and the clustering feature. It will pay dividends, I guarantee it. So what about SQL Server? The requirement here is SQL Server 2014 through 2017. I do want to clarify this a bit too. In other posts and documentation you will see SQL Server 2016 as the requirement. That is only because SQL Server 2016 was current when Windows 2016 released. Since the magic is really in Windows and not in SQL Server, you are supported with SQL 2014 since it is supported on Windows Server 2016.

Design Options

There are several combinations you can have in this configuration. You can have nodes that are not part of a domain at all, or some that are, or some that are joined to completely different domains. Below is a chart with the different combinations. Note that for anything where we are joined to two different domains, we mean complete autonomy. The assumption is that the domains are in completely separate forests and there are no domain trusts in place. Remember that prior to this enhancement, nodes in a cluster had to be in the same domain. Even if the domain was trusted and within the same forest, this would not work.

Options Matrix

Domain A	Domain B	Workgroup A	Workgroup B
Node 1		Node 2
	Node 1	Node 2
Node 1	Node 2
		Node 1	Node 2

Pre-requisites

We do have a few pre-requisites that must be met prior to creating the cluster.

We need local accounts on each node that have identical names and passwords
The local accounts need to be members of the local administrators group
We need to set the LocalAccountTokenFilterPolicy in the registry
- This only needs to be done when using a non-builtin local administrator account
Windows Failover Clustering Feature must be installed
Primary DNS Suffix must be configured

Setup

By now you want to know how to setup and configure this coolness. Today is your lucky day...here is a video walk through! https://www.youtube.com/watch?v=d6adWpf00rs&t=970s

Code

Cluster Setup

[powershell] #First we have to create a local user with the same name and password on each node. #The account is required to be in the local Adminstrators group $Password = Read-Host -AsSecureString New-LocalUser "WSFC" -Password $Password -FullName "WSFC" ` -Description "Used for Domain Independent Cluster" ` -PasswordNeverExpires -UserMayNotChangePassword Add-LocalGroupMember -Group "Users" -Member "WSFC" Add-LocalGroupMember -Group "Administrators" -Member "WSFC" #Next we have to set the LocalAccountTokenFilterPolicy in the registry since we are not using a built-in account new-itemproperty -path HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System -Name LocalAccountTokenFilterPolicy -Value 1 #Next we need to make sure the machine has a primary DNS Suffix #This should NOT be required on domain joined machines since the domain name is automatically set as the PDS #Only run this on workgroup "Non-Domain joined machines" $DNSSuffix = "independent.com" Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\" -Name Domain -Value $DNSSuffix Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\" -Name "NV Domain" -Value $DNSSuffix #We changed the primary DNS Suffix so we need to reboot for the change to take affect Restart-Computer -ComputerName $env:computername -Force #Now we actually create the cluster #For clarity this only gets run on the first node New-Cluster –Name NoDomainClus -Node Server1,Demo2 -AdministrativeAccessPoint DNS ` -StaticAddress 10.0.0.41 -NoStorage -IgnoreNetwork 192.168.1.0/24 [/powershell] Here is the code to create the endpoints and certificates as well as install the certificates. Check out this post from Kevin Farley for the source.

Create Endpoint and Certificate

[sql] /* — This procedure automates the creation of a local certificate and the endpoints required for a domainless AG. — Parameters are the strong password for the cert, and the location of a share which receives the backup of the cert. — The share should be accessible to all nodes in the AG, as they will need to read the certs for each other. — The procedure also creates the endpoint based upon the newly created cert. — https://blogs.technet.microsoft.com/dataplatforminsider/2015/12/15/enhanced-always-on-availability-groups-in-sql-server-2016/ EXEC CreateEndpointCert '\\demo2\temp', '1R3@llyStr0ngP@ssw0rd!' */ DROP PROCEDURE IF EXISTS CreateEndpointCert; GO CREATE PROCEDURE CreateEndpointCert @ShareName SYSNAME , @StrongPassword SYSNAME AS BEGIN --This must be executed in the context of Master IF (DB_NAME() <> 'master') BEGIN PRINT N'This SP must be executed in master. USE master and then retry.' RETURN (-1) END DECLARE @DynamicSQL varchar(1000); DECLARE @CompName varchar(250); DECLARE @HasMasterKey INT; SELECT @CompName = CONVERT(SysName, SERVERPROPERTY('MachineName')); -- Only create a master key if it doesn't already exist SELECT @HasMasterKey = is_master_key_encrypted_by_server from sys.databases where name = 'master' IF (@HasMasterKey = 0) BEGIN --Create a MASTER KEY to encrypt the certificate. SET @DynamicSQL = CONCAT('CREATE MASTER KEY ENCRYPTION BY PASSWORD = ' , QUOTENAME(@StrongPassword, '''')); EXEC (@DynamicSQL) END --Create the certificate to authenticate the endpoint IF EXISTS (SELECT name from sys.certificates WHERE name = QUOTENAME(@CompName + '-Cert')) BEGIN SET @DynamicSQL = CONCAT('DROP CERTIFICATE ', QUOTENAME(@CompName + '-Cert')); EXEC (@DynamicSQL); END SET @DynamicSQL = CONCAT('CREATE CERTIFICATE ', QUOTENAME(@CompName + '-Cert'), ' WITH SUBJECT = ', QUOTENAME(@CompName, '''')) ; EXEC (@DynamicSQL); --Create the database mirroring endpoint authenticated by the certificate. SET @DynamicSQL = CONCAT('CREATE ENDPOINT AlwaysOn_EP STATE = STARTED AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL) FOR DATABASE_MIRRORING (AUTHENTICATION = CERTIFICATE ',QUOTENAME(@CompName + '-Cert'), ' , ENCRYPTION = REQUIRED ALGORITHM AES, ROLE = ALL)'); EXEC (@DynamicSQL); --Back up the certificate to a common network share for import into other nodes in the cluster SET @DynamicSQL = CONCAT('BACKUP CERTIFICATE ',QUOTENAME(@CompName + '-Cert'),' To FILE = ', QUOTENAME( @ShareName + '\SQL-' + @CompName + '.cer', '''')); EXEC (@DynamicSQL); END GO [/sql]

Install Endpoint Certificate

[sql] /* — This procedure assumes that a certificate has been created on another node in the AG, and backed up to a common network share. — Parameters: — @CompName – The name of the computer whose certificate needs to be installed here. i.e. the other replica that this node needs to communicate with. — @ShareName – A common network share to which certificates were backed up from each machine in the cluster/AG. — @StrongPassword – A strong password to be used for the login created to log in on behalf of the endpoint on the other node. — — This procedure assumes that each node has run CreateEndpointCert and that all cert backup files reside on the share pointed to by the second parameter. — The procedure creates a login and a user for the remote machine, and then created a certificate to authorize the user when the certificate is used as authentication from the remote endpoint. -- https://blogs.technet.microsoft.com/dataplatforminsider/2015/12/15/enhanced-always-on-availability-groups-in-sql-server-2016/ EXEC dbo.InstallEndpointCert 'Server1', '\\demo2\temp', '1R3@llyStr0ngP@ssw0rd!' EXEC dbo.InstallEndpointCert 'Demo2', '\\demo2\temp', '1R3@llyStr0ngP@ssw0rd!' */ DROP PROCEDURE IF EXISTS [dbo].[InstallEndpointCert]; GO CREATE PROCEDURE [dbo].[InstallEndpointCert] @CompName SYSNAME, @ShareName SYSNAME, @StrongPassword SYSNAME AS BEGIN DECLARE @DynamicSQL varchar(1000); DECLARE @MyCompName varchar(250); SELECT @MyCompName = CONVERT(SysName, SERVERPROPERTY('MachineName')); --Don't need to create LOGINs for the local system IF (@MyCompName <> @CompName) BEGIN IF EXISTS (SELECT name from sys.certificates WHERE name = QUOTENAME(@CompName + '-Cert')) BEGIN SET @DynamicSQL = CONCAT('DROP CERTIFICATE ', QUOTENAME(@CompName +'-Cert')); EXEC (@DynamicSQL); END SET @DynamicSQL = CONCAT('DROP USER ', QUOTENAME( @CompName + '-User')); EXEC (@DynamicSQL); --SET @DynamicSQL = CONCAT('DROP LOGIN ', QUOTENAME (@CompName + '-Login')); --EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('CREATE LOGIN ', QUOTENAME (@CompName + '-Login'), ' WITH PASSWORD= ', QUOTENAME( @StrongPassword, '''')); EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('CREATE USER ', QUOTENAME( @CompName + '-User'), ' FOR LOGIN ', QUOTENAME(@CompName + '-Login')); EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('CREATE CERTIFICATE ', QUOTENAME(@CompName +'-Cert'), ' AUTHORIZATION ', QUOTENAME(@CompName +'-User'), ' FROM FILE = ', QUOTENAME(@ShareName + '\SQL-' + @CompName + '.cer' , '''')); EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('GRANT CONNECT ON ENDPOINT::AlwaysON_EP TO ', QUOTENAME(@CompName +'-Login')); EXEC (@DynamicSQL); END END GO [/sql]

Availability Group Setup (Run in SQLCMD Mode)

[sql] /* Author: Ryan Adams Website: https://www.ryanjadams.com Twitter: @ryanjadams This script must be executed in SQLCMD mode. This script was designed to setup an AlwaysOn Availability Group in a custom lab environment. Use at your own risk and DO NOT run this in production. Make sure you read and understand it thoroughly. */ /* First we connect to each replica in order and create logins for the account running the SQL Server Service on the other replicas Next we connect to each replica and create the endpoint. We also grant connect permission on the endpoint to all of the other replica service accounts. In a Domain Independent Cluster this must be done using certificates. Use the following scripts to do that. CreateEndpointCert.sql InstallEndpointCert.sql */ /* Here we connect to each replica and ensure that the AlwaysOn extended events session is set to run at startup and that it is also started */ :Connect DEMO2 IF EXISTS(SELECT * FROM sys.server_event_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER WITH (STARTUP_STATE=ON); END IF NOT EXISTS(SELECT * FROM sys.dm_xe_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER STATE=START; END GO :Connect SERVER1 -U MyAdmin -P NotGettingThisPassword IF EXISTS(SELECT * FROM sys.server_event_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER WITH (STARTUP_STATE=ON); END IF NOT EXISTS(SELECT * FROM sys.dm_xe_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER STATE=START; END GO /* There will be 1 database included in our AG and it currently exists on DEMO2. Prior to SQL 2016 we would have to do the following: Backup each database data file to a share and then restore it on each replica making sure to use the "NORECOVERY" clause Backup each database log file to a share and then restore it on each replica making sure to use the "NORECOVERY" clause With SQL 2016 we can use Direct Seeding instead. */ /* Here we connect to our primary replica (DEMO2) and create our AG. */ :Connect DEMO2 /* We can use trace flag 9567 to enable compression for the VDI backup for the seeding process */ DBCC TRACEON (9567, -1); GO USE [master]; GO CREATE AVAILABILITY GROUP [MyAG] WITH (AUTOMATED_BACKUP_PREFERENCE = SECONDARY, FAILURE_CONDITION_LEVEL = 3, HEALTH_CHECK_TIMEOUT = 30000, DB_FAILOVER = ON, DTC_SUPPORT = NONE) /* You can also use SECONDARY_ONLY, PRIMARY, or NONE for backup preference Failure condition levels are from the least restrcitive of 1(SQL Service is down) to 5(Exhaustion of worker threads). 3 is the default. Health check default is 30 seconds, represented in milliseconds. This is how long we wait for sp_server_diagnostics to return. */ FOR DATABASE [DomainIndependent] REPLICA ON N'DEMO2' WITH (ENDPOINT_URL = N'TCP://DEMO2.independent.com:5022', FAILOVER_MODE = AUTOMATIC, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT, BACKUP_PRIORITY = 50, SECONDARY_ROLE(ALLOW_CONNECTIONS = Read_Only), SESSION_TIMEOUT = 10, SEEDING_MODE = AUTOMATIC), N'SERVER1' WITH (ENDPOINT_URL = N'TCP://SERVER1.americans.com:5022', FAILOVER_MODE = AUTOMATIC, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT, BACKUP_PRIORITY = 60, SECONDARY_ROLE(ALLOW_CONNECTIONS = Read_Only), SESSION_TIMEOUT = 10, SEEDING_MODE = AUTOMATIC); GO /* Here we create a listener for our AG */ ALTER AVAILABILITY GROUP [MyAG] ADD LISTENER N'MyAG' ( WITH IP ((N'10.0.0.11', N'255.255.255.0')) , PORT=1433); GO /* Now that the AG exists we connect to each secondary replica and join it to the group We also have to grant the CREATE ANY DATABASE permission so Seeding can create the DBs */ :Connect SERVER1 -U MyAdmin -P NotGettingThisPassword ALTER AVAILABILITY GROUP [MyAG] JOIN; GO ALTER AVAILABILITY GROUP [MyAG] GRANT CREATE ANY DATABASE; GO /* This query let's us view the progress of the seeding operation SELECT start_time, ag.name, db.database_name, current_state, performed_seeding, failure_state, failure_state_desc FROM sys.dm_hadr_automatic_seeding autos JOIN sys.availability_databases_cluster db ON autos.ag_db_id = db.group_database_id JOIN sys.availability_groups ag ON autos.ag_id = ag.group_id; GO SELECT * FROM sys.dm_hadr_physical_seeding_stats; GO */ /* Prior to SQL 2016 we had to connect to each secondary replica and add our DBs to the group. With seeding we no longer have to do this as it is joined during the seeding process :Connect SERVER1 -U MyAdmin -P NotGettingThisPassword ALTER DATABASE [MyAG_DB1] SET HADR AVAILABILITY GROUP = [MyAG]; ALTER DATABASE [MyAG_DB2] SET HADR AVAILABILITY GROUP = [MyAG]; ALTER DATABASE [MyAG_DB3] SET HADR AVAILABILITY GROUP = [MyAG]; GO */ /* Now we need to turn our trace flag back off */ :Connect DEMO2 DBCC TRACEOFF (9567, -1); GO [/sql]

Resources

You might also be interested in Read-Scale or Clusterless Availability Groups. The other thing most folks find challenging with Availability Groups are how to sync users and server level objects. Check out Sync SQL Logins for more information.

Read-Scale Availability Group Setup

Ryan J. Adams — Thu, 05 Jul 2018 08:00:34 GMT

What is a Read-Scale Availability Group?

A Read-Scale Availability Group is a Clusterless Availability Group. It's sole purpose and design is to scale out a read workload. More importantly is what it is not. It is NOT a High Availability solution. Since this design has no cluster under it, you lose things like automatic failover and database level health detection. An example use case is maybe you have reports that run for customers that are in your DMZ that is fire-walled off from your internal network. Opening up ports for Active Directory so that you can have a cluster means opening a ton of ephemeral ports and ports with high attack vectors. Remember the Slammer worm? This solution removes those dependencies.

Requirements

What you need to create this solution is pretty simple. You need Windows Server 2016 or greater. You also need SQL Server 2017 or greater.

Pre-requisites

We do have a few pre-requisites that must be met.

We need local accounts on each node that have identical names and passwords
The local accounts need to be members of the local administrators group
We need to set the LocalAccountTokenFilterPolicy in the registry
- This only needs to be done when using a non-builtin local administrator account
Primary DNS Suffix must be configured

Setup

Here is a video where I walk through creating a Read-Scale or Clusterless Availability Group. https://www.youtube.com/watch?v=UdB8G9YCxgQ

Code

All of the code below is what I used in the above video in my test environment. You should test this code prior to using it and you assume all responsibility as it is only provided for demonstration purposes. You will need to run this on each replica you plan to add to your Availability Group. [powershell] #First we have to create a local user with the same name and password on each node. #The account is required to be in the local Adminstrators group $Password = Read-Host -AsSecureString New-LocalUser "WSFC" -Password $Password -FullName "WSFC" ` -Description "Used for Domain Independent Cluster" ` -PasswordNeverExpires -UserMayNotChangePassword Add-LocalGroupMember -Group "Users" -Member "WSFC" Add-LocalGroupMember -Group "Administrators" -Member "WSFC" #Next we have to set the LocalAccountTokenFilterPolicy in the registry since we are not using a built-in account new-itemproperty -path HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System -Name LocalAccountTokenFilterPolicy -Value 1 #Next we need to make sure the machine has a primary DNS Suffix #This should NOT be required on domain joined machines since the domain name is automatically set as the PDS #Only run this on workgroup "Non-Domain joined machines" $DNSSuffix = "independent.com" Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\" -Name Domain -Value $DNSSuffix Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\" -Name "NV Domain" -Value $DNSSuffix #We changed the primary DNS Suffix so we need to reboot for the change to take affect Restart-Computer -ComputerName $env:computername -Force [/powershell] Run this on each replica to create an endpoint that is protected by a certificate and backs the certificate up to a share. The original code came from Kevin Farlee and can be found in THIS POST. [sql] /* — This procedure automates the creation of a local certificate and the endpoints required for a domainless AG. — Parameters are the strong password for the cert, and the location of a share which receives the backup of the cert. — The share should be accessible to all nodes in the AG, as they will need to read the certs for each other. — The procedure also creates the endpoint based upon the newly created cert. — https://blogs.technet.microsoft.com/dataplatforminsider/2015/12/15/enhanced-always-on-availability-groups-in-sql-server-2016/ EXEC CreateEndpointCert '\\demo2\temp', '1R3@llyStr0ngP@ssw0rd!' */ DROP PROCEDURE IF EXISTS CreateEndpointCert; GO CREATE PROCEDURE CreateEndpointCert @ShareName SYSNAME , @StrongPassword SYSNAME AS BEGIN --This must be executed in the context of Master IF (DB_NAME() <> 'master') BEGIN PRINT N'This SP must be executed in master. USE master and then retry.' RETURN (-1) END DECLARE @DynamicSQL varchar(1000); DECLARE @CompName varchar(250); DECLARE @HasMasterKey INT; SELECT @CompName = CONVERT(SysName, SERVERPROPERTY('MachineName')); -- Only create a master key if it doesn't already exist SELECT @HasMasterKey = is_master_key_encrypted_by_server from sys.databases where name = 'master' IF (@HasMasterKey = 0) BEGIN --Create a MASTER KEY to encrypt the certificate. SET @DynamicSQL = CONCAT('CREATE MASTER KEY ENCRYPTION BY PASSWORD = ' , QUOTENAME(@StrongPassword, '''')); EXEC (@DynamicSQL) END --Create the certificate to authenticate the endpoint IF EXISTS (SELECT name from sys.certificates WHERE name = QUOTENAME(@CompName + '-Cert')) BEGIN SET @DynamicSQL = CONCAT('DROP CERTIFICATE ', QUOTENAME(@CompName + '-Cert')); EXEC (@DynamicSQL); END SET @DynamicSQL = CONCAT('CREATE CERTIFICATE ', QUOTENAME(@CompName + '-Cert'), ' WITH SUBJECT = ', QUOTENAME(@CompName, '''')) ; EXEC (@DynamicSQL); --Create the database mirroring endpoint authenticated by the certificate. SET @DynamicSQL = CONCAT('CREATE ENDPOINT AlwaysOn_EP STATE = STARTED AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL) FOR DATABASE_MIRRORING (AUTHENTICATION = CERTIFICATE ',QUOTENAME(@CompName + '-Cert'), ' , ENCRYPTION = REQUIRED ALGORITHM AES, ROLE = ALL)'); EXEC (@DynamicSQL); --Back up the certificate to a common network share for import into other nodes in the cluster SET @DynamicSQL = CONCAT('BACKUP CERTIFICATE ',QUOTENAME(@CompName + '-Cert'),' To FILE = ', QUOTENAME( @ShareName + '\SQL-' + @CompName + '.cer', '''')); EXEC (@DynamicSQL); END GO [/sql] This will install the certificate of the other replicas on the replica where you run it. This will need to be run on each replica and run once for every other replica in the Availability Group. If we have Node 1 and Node 2 then we are installing the Node 1 certificate on Node 2 and vice versa. [sql] /* — This procedure assumes that a certificate has been created on another node in the AG, and backed up to a common network share. — Parameters: — @CompName – The name of the computer whose certificate needs to be installed here. i.e. the other replica that this node needs to communicate with. — @ShareName – A common network share to which certificates were backed up from each machine in the cluster/AG. — @StrongPassword – A strong password to be used for the login created to log in on behalf of the endpoint on the other node. — — This procedure assumes that each node has run CreateEndpointCert and that all cert backup files reside on the share pointed to by the second parameter. — The procedure creates a login and a user for the remote machine, and then created a certificate to authorize the user when the certificate is used as authentication from the remote endpoint. -- https://blogs.technet.microsoft.com/dataplatforminsider/2015/12/15/enhanced-always-on-availability-groups-in-sql-server-2016/ EXEC dbo.InstallEndpointCert 'Server1', '\\demo2\temp', '1R3@llyStr0ngP@ssw0rd!' EXEC dbo.InstallEndpointCert 'Demo2', '\\demo2\temp', '1R3@llyStr0ngP@ssw0rd!' */ DROP PROCEDURE IF EXISTS [dbo].[InstallEndpointCert]; GO CREATE PROCEDURE [dbo].[InstallEndpointCert] @CompName SYSNAME, @ShareName SYSNAME, @StrongPassword SYSNAME AS BEGIN DECLARE @DynamicSQL varchar(1000); DECLARE @MyCompName varchar(250); SELECT @MyCompName = CONVERT(SysName, SERVERPROPERTY('MachineName')); --Don't need to create LOGINs for the local system IF (@MyCompName <> @CompName) BEGIN IF EXISTS (SELECT name from sys.certificates WHERE name = QUOTENAME(@CompName + '-Cert')) BEGIN SET @DynamicSQL = CONCAT('DROP CERTIFICATE ', QUOTENAME(@CompName +'-Cert')); EXEC (@DynamicSQL); END SET @DynamicSQL = CONCAT('DROP USER ', QUOTENAME( @CompName + '-User')); EXEC (@DynamicSQL); --SET @DynamicSQL = CONCAT('DROP LOGIN ', QUOTENAME (@CompName + '-Login')); --EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('CREATE LOGIN ', QUOTENAME (@CompName + '-Login'), ' WITH PASSWORD= ', QUOTENAME( @StrongPassword, '''')); EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('CREATE USER ', QUOTENAME( @CompName + '-User'), ' FOR LOGIN ', QUOTENAME(@CompName + '-Login')); EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('CREATE CERTIFICATE ', QUOTENAME(@CompName +'-Cert'), ' AUTHORIZATION ', QUOTENAME(@CompName +'-User'), ' FROM FILE = ', QUOTENAME(@ShareName + '\SQL-' + @CompName + '.cer' , '''')); EXEC (@DynamicSQL); SET @DynamicSQL = CONCAT('GRANT CONNECT ON ENDPOINT::AlwaysON_EP TO ', QUOTENAME(@CompName +'-Login')); EXEC (@DynamicSQL); END END GO [/sql] This code will create the Availability Group. It must be run in SQLCMD mode. [sql] /* Author: Ryan Adams Website: https://www.ryanjadams.com Twitter: @ryanjadams This script must be executed in SQLCMD mode. This script was designed to setup an AlwaysOn Availability Group in a custom lab environment. Use at your own risk and DO NOT run this in production. Make sure you read and understand it thoroughly. */ /* First we connect to each replica in order and create logins for the account running the SQL Server Service on the other replicas Next we connect to each replica and create the endpoint. We also grant connect permission on the endpoint to all of the other replica service accounts. In a clusterless AG this must be done using certificates. Use the following scripts to do that. CreateEndpointCert.sql InstallEndpointCert.sql */ /* Here we connect to each replica and ensure that the AlwaysOn extended events session is set to run at startup and that it is also started */ :Connect DEMO2 IF EXISTS(SELECT * FROM sys.server_event_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER WITH (STARTUP_STATE=ON); END IF NOT EXISTS(SELECT * FROM sys.dm_xe_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER STATE=START; END GO :Connect SERVER1 -U MyAdmin -P NotGettingThisPassword IF EXISTS(SELECT * FROM sys.server_event_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER WITH (STARTUP_STATE=ON); END IF NOT EXISTS(SELECT * FROM sys.dm_xe_sessions WHERE name='AlwaysOn_health') BEGIN ALTER EVENT SESSION [AlwaysOn_health] ON SERVER STATE=START; END GO /* There will be 1 database included in our AG and it currently exists on DEMO2. Prior to SQL 2016 we would have to do the following: Backup each database data file to a share and then restore it on each replica making sure to use the "NORECOVERY" clause Backup each database log file to a share and then restore it on each replica making sure to use the "NORECOVERY" clause With SQL 2016 and above we can use Direct Seeding instead. */ /* Here we connect to our primary replica (DEMO2) and create our AG. Note the CLUSTER_TYPE = None and the FAILOVER_MODE = MANUAL */ :Connect DEMO2 /* We can use trace flag 9567 to enable compression for the VDI backup for the seeding process */ DBCC TRACEON (9567, -1); GO USE [master]; GO CREATE AVAILABILITY GROUP [MyAG] WITH (AUTOMATED_BACKUP_PREFERENCE = SECONDARY, FAILURE_CONDITION_LEVEL = 3, HEALTH_CHECK_TIMEOUT = 30000 , DB_FAILOVER = ON, DTC_SUPPORT = NONE, CLUSTER_TYPE = None) /* You can also use SECONDARY_ONLY, PRIMARY, or NONE for backup preference Failure condition levels are from the least restrcitive of 1(SQL Service is down) to 5(Exhaustion of worker threads). 3 is the default. Health check default is 30 seconds, represented in milliseconds. This is how long we wait for sp_server_diagnostics to return. */ FOR DATABASE [DomainIndependent] REPLICA ON N'DEMO2' WITH (ENDPOINT_URL = N'TCP://DEMO2.independent.com:5022', FAILOVER_MODE = MANUAL, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT, BACKUP_PRIORITY = 50, SECONDARY_ROLE(ALLOW_CONNECTIONS = Read_Only), SESSION_TIMEOUT = 10, SEEDING_MODE = AUTOMATIC), N'SERVER1' WITH (ENDPOINT_URL = N'TCP://SERVER1.americans.com:5022', FAILOVER_MODE = MANUAL, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT, BACKUP_PRIORITY = 60, SECONDARY_ROLE(ALLOW_CONNECTIONS = Read_Only), SESSION_TIMEOUT = 10, SEEDING_MODE = AUTOMATIC); GO /* Now that the AG exists we connect to each secondary replica and join it to the group We also have to grant the CREATE ANY DATABASE permission so Seeding can create the DBs Notice the "WITH(CLUSTER_TYPE = NONE)" clause that is new */ :Connect SERVER1 -U MyAdmin -P NotGettingThisPassword ALTER AVAILABILITY GROUP [MyAG] JOIN WITH (CLUSTER_TYPE = NONE); GO ALTER AVAILABILITY GROUP [MyAG] GRANT CREATE ANY DATABASE; GO /* This query let's us view the progress of the seeding operation SELECT start_time, ag.name, db.database_name, current_state, performed_seeding, failure_state, failure_state_desc FROM sys.dm_hadr_automatic_seeding autos JOIN sys.availability_databases_cluster db ON autos.ag_db_id = db.group_database_id JOIN sys.availability_groups ag ON autos.ag_id = ag.group_id; GO SELECT * FROM sys.dm_hadr_physical_seeding_stats; GO */ /* Prior to SQL 2016 we had to connect to each secondary replica and add our DBs to the group. With seeding we no longer have to do this as it is joined during the seeding process :Connect SERVER1 -U MyAdmin -P NotGettingThisPassword ALTER DATABASE [MyAG_DB1] SET HADR AVAILABILITY GROUP = [MyAG]; ALTER DATABASE [MyAG_DB2] SET HADR AVAILABILITY GROUP = [MyAG]; ALTER DATABASE [MyAG_DB3] SET HADR AVAILABILITY GROUP = [MyAG]; GO */ /* Now we need to turn our trace flag back off */ :Connect DEMO2 DBCC TRACEOFF (9567, -1); GO [/sql]

Resources

This solution is designed to scale out reads using an Availability Group without the requirement of a cluster. As discussed at the beginning of the article, you lose some things. What about Read-Only Routing, though? You can still use this, but the configuration is a bit different. To learn how to configure Read-Only Routing in a Read-Scale Availability Group read THIS POST by Sean Gallardy. You might also be interested in Domain Independent Availability Groups. That architecture retains the use of a cluster, but the nodes no longer have to be in a domain or they can even be in different domains. Read Domain Independent Clusters for more information. The other thing most folks find challenging with Availability Groups are how to sync users and server level objects. Check out Sync SQL Logins for more information.

MSDTC Supported Configurations

Ryan J. Adams — Thu, 05 Jul 2018 08:00:24 GMT

The MSDTC Configuration Conundrum

MSDTC configuration is not as straight forward as you might think. It's a different choice if you are using a local MSDTC, clustered MSDTC, on-premises, Azure, Failover Cluster, or Availability Group. Every one of those variables leads you down a different path with different choices. The goal of this article is to clarify those choices. If you think you do not use the MSDTC, you better check again. I frequently see folks using it that do not realize it. The most common is Linked Servers. You got it. If you are using Linked Servers, you're using the MSDTC. You are no longer required to install an MSDTC with a SQL Failover Cluster starting in SQL Server 2008. That does not mean you do not need an MSDTC configuration. If you wanted to make your application database highly available, but your application did not use the MSDTC you were being required to install one anyway. That limitation is what was removed, not that you simply didn't need a clustered MSDTC.

How it Works

Let's start with a simple description of the purpose of the MSDTC. First the application starts a transaction that queries two different servers. That's a distributed transaction. Since parts of the query are going to two different locations, something has to monitor it to ensure transaction consistency. That's the job of the MSDTC. What we do not want is for the transaction to go make updates to the first destination and commit that without making sure the updates to the second destination also commit. If either one of them fails we want to roll it back and if they succeed we want to commit them. The MSDTC logs the transactions in its own log so it can track them. This is key to remember. It's the availability of this log that we care about.

Stand-alone

We will continue with a stand-alone server to see how this works and then we'll work through Failover Cluster Instances and Availability Groups. On a stand-alone all we have is the local MSDTC. When our application runs a transaction that inserts rows in a database on this server and also on another server, the local MSDTC enlists in the transaction. Its job is to log the transactions and ensure any in-doubt transactions are either aborted (rolled back) or committed (rolled forward). There is not much ambiguity in this scenario since it is not designed to be highly available, but it gives us a basis on which to build. [caption id="attachment_2018" align="aligncenter" width="262"] Local MSDTC Configuration on Stand-alone Server[/caption]

Failover Cluster

Here is where things get a bit more interesting. We have a choice to use either a local MSDTC or a clustered MSDTC. Using a clustered MSDTC configuration is straightforward. If Node1 fails, both the SQL Failover Cluster Instance and the Clustered MSDTC fail over to Node2. Since the Clustered MSDTC is using shared storage, it still has its log and can handle any in-doubt transactions. [caption id="attachment_2020" align="aligncenter" width="300"] FCI with a Clustered MSDTC Configuration[/caption] What happens if we choose a Local MSDTC? This will mean that your MSDTC is not highly available, but it's more available than you might think. If Node1 fails and the local MSDTC on that node has an in-doubt transaction, the local MSDTC on Node2 kicks in after the failover. The local MSDTC on Node2 tries to contact the local MSDTC on Node1 and retrieve the status of any in-doubt transactions. This process is attempted 3 times. If the old primary (Node1) still cannot be contacted, SQL takes its direction from the In-doubt Transaction Configuration or "in-doubt xact resolution" in sp_configure. [caption id="attachment_2021" align="aligncenter" width="300"] FCI with a Local MSDTC Configuration[/caption] You can configure the In-doubt Transaction Configuration for either Abort(2) which rolls the transactions back, or Commit(1) which rolls the transactions forward. If this setting is not configured (default) then the database is put into suspect mode. You can manually resolve the in-doubt transactions using the KILL command and bring the database online. See this article for specific instructions for this scenario. [caption id="attachment_2022" align="aligncenter" width="521"] In-doubt Transaction Flow When Using a Local MSDTC[/caption]

Availability Group

The MSDTC works exactly the same in an Availability Group as it does in a Failover Cluster Instance, with a couple of distinctions. The first is on what versions it is supported. The versions also differ in the features they support. Here is what that looks like.

No support in Availability Groups until SQL 2016
SQL 2016
- Cross instance transactions are supported
- Cross database transactions within same SQL AG Instance are NOT supported
SQL 2016 SP2
- Cross instance transactions are supported
- Cross database transactions within same SQL AG Instance are supported
SQL 2017
- Cross instance transactions are supported
- Cross database transactions within same SQL AG Instance are supported

The second distinction is that you need to create the availability group using the WITH clause of DTC_SUPPORT = PER_DB. In SQL Server 2017 and SQL Server 2016 SP2 you can now use the ALTER statement to change this after AG creation. Read here for details on its usage. What's important about this distinction is what it means under the hood. In a Failover Cluster Instance, the instance itself becomes the Resource Manager, but in an Availability Group each database becomes its own Resource Manager. Remember that in an AG we have completely different instances of SQL running on each replica so we cannot use an instance level Resource Manager as the ID would be different. The RMID comes over with the database when it fails over allowing it to resolve in-doubt transactions. An FCI is a single instance so we don't have this problem as the RMID is consistent across failovers. Read this to get a better understanding of the role of the Resource Manager.

Availability Group Local MSDTC versus Clustered MSDTC

We created database DB1 in an Availability Group with the primary on Node1 and secondary on Node2
We did some DTC transactions and Node1 went down. There were some in-doubt transactions. DB1 will fail over to Node2.
If using a local DTC:
- What really happens
  - SQL Server uses agile recovery to recover in-doubt transactions. The Agile Recovery concept actually happens at recovery time, when the resource manager provides the PrepareInfo back to the MSDTC it is connected to at recovery time. If the PrepareInfo provided by the resource manager at recovery time indicates that the PrepareInfo came from a DIFFERENT MSDTC than the MSDTC the RM is currently connected to, the MSDTC proxy will connect to the “original” MSDTC “under the covers” to find out the outcome of the transaction. So, if an RM is connected to DTC1 at Prepare time, the PrepareInfo identifies DTC1. If the RM is connected to DTC2 at Recovery time, the DTC proxy will ALSO connect to DTC1 to find out the outcome because DTC2 doesn’t know anything about the transaction.
- An easier way to think about it
  - SQL Server on Node2 asks MSDTC on Node2 about the outcome for the transactions. MSDTC on Node2 figures out that the MSDTC on Node1 knows the outcome of the transaction from the prepare info and hence MSDTC on Node2 asks MSDTC on Node1 for the outcome
  - If MSDTC on Node1 is not available then MSDTC on Node2 cannot get the outcome until MSDTC on Node1 is available (because these outcomes are present in the MSDTC log). At this point SQL follows the In-doubt Transaction Configuration setting. The database will be put in SUSPECT mode if the setting is not configured
If using a clustered MSDTC:
- SQL directly connects to the clustered MSDTC (as this also would have failed over to Node2) to recover the outcome

Azure

In Azure IaaS, the functionality described above does not change. What gets more complicated is the storage for a clustered MSDTC, as it requires the storage to be shared. Microsoft has partners that offer options for shared storage in Azure, which include solutions like iSCSI among others. We also have a native Microsoft solution when using Storage Spaces Direct (S2D). Although you can use this for SQL Failover Cluster Instance Resources in Windows Server 2016, you cannot use it for the MSDTC Resource until Windows Server 2016 Release Build Version 1709. Windows Server 2016 Release Build Version 1709 and 1803 are core builds and do not have the Desktop Experience (GUI). Windows Server 2019 is the first build to support this functionality and also contain the Desktop Experience. Read here for more on builds that support the MSDTC on S2D in Azure. That article includes a video walk through of building the entire solution with an FCI on a Windows Cluster with the MSDTC on S2D in an Azure VM. Another difference in Azure is outside of the servers and MSDTC, but is required to make it work properly. A clustered MSDTC has a network name and in Azure we need an internal load balancer to direct that traffic. This is important because you need to use a standard load balancer (not basic) which in turn means you need to use standard IP addresses. You will need to keep that in mind when building your VMs. Make sure to check out the article in the above paragraph as it describes and demonstrates this configuration.

MSDTC Throughput

I do want to mention one other noteworthy thing to think about as you design your architecture. If you have multiple instances of SQL Server running and you create one clustered MSDTC in its own resource group, every instance uses that same MSDTC. If you have a lot of distributed transactions this can become a bottleneck. The other issue that can get introduced with a single clustered MSDTC in its own role, is that you cannot guarantee it will run on the same node as your SQL instance. You could easily have an instance running on Node2 and using an MSDTC running on Node1. This can introduce network latency. Refer back to the second diagram in this post titled, "FCI with a Clustered MSDTC Configuration" and imagine the SQL Resource is running on Node2 while the MSDTC Resource remains on Node1. The best practice is to have 1 MSDTC per instance and put it in the same role as the instance. This makes sure the MSDTC for that instance is running on the same node and spreads the load out. Here is how that would look with two instances of SQL Server and each of them having their own MSDTC. [caption id="attachment_2029" align="aligncenter" width="300"] MSDTC Per SQL Instance[/caption]

MSDTC Precedence

It is wise to verify which MSDTC service each SQL instance is using. Below is the precedence in which SQL Server chooses which MSDTC it will use. Note that SQL Server makes this decision at startup. If you make a change, you have to restart SQL Server for it to take effect. You can manually map an instance of SQL Server to an instance of the MSDTC, and the first instance installed of the MSDTC is considered the default instance.

MSDTC in same role
Mapped Instance of MSDTC
Cluster Default MSDTC Instance
Local MSDTC

Support Matrix

This matrix should help guide you as you design your architecture and decide on versions and features.

Configuration	Cross Instance	Cross Database	Local MSDTC	Clustered MSDTC
SQL 2014 and Below AG	MSDTC Not Supported	MSDTC Not Supported	MSDTC Not Supported	MSDTC Not Supported
SQL 2016 AG	Yes	No	Yes*	Yes
SQL 2016 SP2 AG	Yes	Yes	Yes*	Yes
SQL 2017 AG	Yes	Yes	Yes*	Yes

SQL 2016 FCI	Yes	Yes	Yes*	Yes
SQL 2017 FCI	Yes	Yes	Yes*	Yes

Azure VM FCI	=SQL Version	=SQL Version	No Load Balancer	Standard Load Balancer, Shared Storage, Windows Build 1709 or Greater
Azure VM AG	=SQL Version	=SQL Version	No Load Balancer	Standard Load Balancer, Shared Storage, Windows Build 1709 or Greater

Legend Yes* ----> In-doubt Transaction Manager Policy

Migrating Symmetric Keys Between SQL 2016 and SQL 2017/Azure

Sean Gallardy — Fri, 08 Jun 2018 13:55:52 GMT

There has been a recent discussion about migrating symmetric keys between instances and version of SQL Server. It doesn't seem to be a widely known process and there have been some changes between versions.

Background

There are many ways to migrate between versions of SQL Server, some of these ways have more moving parts than others. Sometimes unforeseen issues arise in our given migration methods, one of these might have to do with symmetric key encryption. Symmetric keys in SQL Server are not exportable, so we can't back them up and restore them outside of the database backup or restore. There is, however, a way to create a duplicate key in another database if we know the inputs to the KEY_SOURCE, ALGORITHM, and IDENTITY_VALUE options. Unfortunately, if these are unknown or lost there is no way to recreate the same key. Migrating to SQL Server 2017 Like many others, look at the list of breaking changing is probably one of the first actions completed when looking to upgrade. One of the changes made in SQL Server 2017 was the algorithm used to compute hashes of some of the input values. Since the hash output will change, the actual key will also change. This means you cannot create a duplicate key between SQL Server 2012-16 and SQL Server 2017. It is still possible to complete a database backup and restore operation to migrate the database with the key, but generating a new one will not work. The same issue will arise when migrating to Azure SQL DB.

Solution

If you need to create a duplicate symmetric key on Azure SQL DB, please open a support ticket explaining such. The creation of a duplicate symmetric key can be done through support. If you want to create a duplicate symmetric key on SQL Server 2017 and you have the requisite input information, use trace flag 4631 (requires 2017 CU2). This trace flag can be used to change the hashing algorithm back to the 2012-2016 algorithm and create a duplicate key. Since this trace flag is instance wide, only enable it for the small window of time needed to duplicate they key then turn it off immediately after.

Monitoring Azure Analysis Services with Log Analytics and Power BI

Christopher M Schmidt — Mon, 02 Apr 2018 10:00:53 GMT

How do you monitor Azure Analysis Services? How many users are connected and who are they? These are great questions to understand around your AAS environment. While metric data is exposed via the Metrics blade for Azure Analysis Services in the portal, it's a quick means to answer these questions. What if you wanted to combine this information with other operational data within your organization? Especially around the area of QPU (or query processing units, which is how AAS is priced). While extended events work in Azure Analysis Services, parsing the resulting XML files into human readable form is cumbersome, difficult, and time consuming. Just as many aspects of the cloud should require some thought process into how systems are designed, it's prudent to rethink the monitoring aspect as well. By using Microsoft Log Analytics, it's possible to build a complete monitoring solution for Azure Analysis Services with a simple Power BI Dashboard that can be accessed along with the rest of needed operational information for system administrators. Log Analytics provides near real-time data exploration for telemetry. There is a great reference here from Brian Harry. The remainder of this post serves to detail setting up this process. One of the elegant approaches of this solution is that once it is set up, apart from the refresh of the power bi report no maintenance is required between the azure analysis services and monitoring step (as opposed to extended events). Without further delay, let's look at the steps required.

Create a Log Analytics instance in your Azure subscription
Configure Azure Analysis Services to send event and performance data to the log analytics instance
Using Log Analytics as the data source, report on the data from Power BI

As the first step in the process, the first item we need to create is a Log Analytics instance, which is a part of Microsoft OMS (Operations Management Suite). For anyone unfamiliar, OMS is essentially Bing for your telemetry analytics data. For more information, see this "What is Log Analytics" page on the Microsoft docs. In the Azure portal, simply select "Create a resource", and then type Log Analytics: After clicking Create, select to either use an existing OMS workspace or to link to an existing one within the tenant. Choose to either use an existing resource group or create a new one, and then specify a location. Finally, select the pricing tier and click Ok: Once this is complete, you'll be presented with the OMS workspace node, which is essentially just a summary page. For now, just leave it as is. Next up we'll configure the magic. One of the major benefits of OMS is the ability to configure both on prem machines to forward information to Log Analytics, or to configure PAAS applications to transmit data, and what's what we'll do here. The doc is relatively straightforward, and the only issue that I encountered when setting it up was that my subscription did not have the Microsoft.insights namespace registered for my subscription. Below is the complete PowerShell Script that I ran against my sub utilizing the script outlined in the Microsoft docs: [code lang="powershell"] ##followed the blog post outlined here: https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-collect-azurepass-posh ##save the azurermdiagnostic script locally that is saved on the powershell gallery. #save-script -Name Enable-AzureRMDiagnostics -Path "C:\PowerShellScripts\" ##create a connection to azure #Login-AzureRmAccount ##install any modules that don't already exist Install-Module AzureRm.Insights Install-Module AzureRM.OperationalInsights Install-Module AzureRM.Resources Install-Module AzureRM.Profile ##check and see the current state of the provider namespace for microsoft insights #Get-AzureRMResourceProvider -ProviderNamespace Microsoft.Insights ##register it if needed #Register-AzureRMResourceProvider -ProviderNamespace Microsoft.insights ##run the saved script from earlier C:\PowerShellScripts\Enable-AzureRMDiagnostics.ps1 #Step 1. select the subscripton where the paas offering resides #Step 2. SElect the resource type that you wanto to send to azure. in this case, find the resource type for Microsoft.AnalysisServices/servers #step 3. Provide the  category of logs to process or type ALL. #allow the script to run. [/code] While running the script, note when it asks for the category of logs to track. For Azure AS, there are 2 options: Engine and Service. The Engine category is what traditionally would be tracked under either profiler or extended events, and the Service category is used for monitoring and tracking traditional perfmon counters (aka metrics). Note that this "shipping" to Log Analytics is not instant. There can be a delay from what I've seen of anywhere from 5-20 minutes for the data to arrive. However what we are now presented with is a hassle free approach to analysis services monitoring! Next up is getting the data out of Log Analytics for reporting. Let's do that now. Back in the Azure portal, in the Log Analytics resource that was created earlier, open up the Overview Blade and then select Analytics: Once this window is open, the first screen presented to you is the Home Page which talks about getting started with AQL. Over on the left hand side of the screen notice the active pane, which contains a root of the oms workspace that you originally created and then a node for "log management". This is where your Azure AS data is being sent. Expanding that node, you see a list of all the tables that are a part of the log management object for Log Analytics. There are two primary tables we are interested in: AzureDiagnostics, which is where Azure AS sends the extended event data (aka engine); and AzureMetrics, which contains the metric data (aka service). In Log Analytics, the queries are built using a language native to the product. For example, to see everything in the AzureDiagnostics table just drop the table into the query window: I've gone ahead and created two queries to pull the data out of the two tables: [code lang="sql"] //AASEvents: AzureDiagnostics | extend LocalTime = TimeGenerated -4h | order by LocalTime desc [/code] [code lang="sql"] //AASMetrics: AzureMetrics | extend LocalTime = TimeGenerated -4h | where ResourceProvider == "MICROSOFT.ANALYSISSERVICES" | order by LocalTime desc [/code] Running these queries results in the data that AAS has sent showing on screen. The next step is to export the data to Power BI. Fortunately Log Analytics again makes this easy: The resulting file that downloads is a custom query that I can simply copy/paste into PowerBI! How cool is that! It also contains the header, credential information, and my query for me to pull in. After generating both of these for the two queries switch over and open up Power BI Desktop and paste them in via the provided instructions. Now as you follow the instructions and create your dashboard, don't forget about perhaps creating some additional insight by using the metrics descriptions that are published on the Microsoft docs page. You could simply point Power BI to this page and have it extract, however if you decide to go that route remember that Power BI does not currently at the time of this writing allow you to mix both online and "on-prem" data sources together for refresh in a single dataset. I chose the copy/paste method for my report. After creating the report, I wound up with something like the below. In a real world scenario, there should be some additional slicers to see by resource group and instance as well as what I have here. The sheer amount of information at your disposal regarding Azure Analysis Services performance is overwhelming. For this use case the interest was simply in who is running the queries and what kinds of queries are they running, but all of the information is available to perform in-depth performance tuning and analysis of the underlying engine, query tuning, etc. By scheduling periodic refreshes of the dataset within Power BI as well, we have an always up to date dashboard of usage in Azure Analysis Services. Consider some of the following scenarios that this dashboard could be used for:

Dynamically pause/restart azure analysis services during times when there is no QPU activity
Dynamically scale up/down an azure analysis services instance based on the time of day and how many QPU units are needed

All of these scenarios could be accomplished using the Azure Analysis Services PowerShell cmdlets.