Chris Mitchell https://docs.microsoft.com/archive/blogs/chmitch/feed.xml Data and Analytics Insights Wed, 17 Apr 2019 03:21:03 GMT en-US hourly 1 Complete guide to setting up Power BI connecting to Postgres w/ refresh enabled. https://docs.microsoft.com/archive/blogs/chmitch/complete-guide-to-setting-up-power-bi-connecting-to-postgres-w-refresh-enabled Mon, 04 Jun 2018 15:09:09 GMT https://blogs.msdn.microsoft.com/chmitch/?p=375
    •   Since there are a ton of steps here, I've broken it into a few logical parts. This keeps things better organized and helps you skip over sections that are irrelevant to your scenario. The parts are:
        1. Part 1 - Provision your Azure Postgres Database and install the components for connecting to Postgres
        2. Part 2 - Register the appropriate certificate so SSL connections to Postgres work.
        3. Part 3 - Connect to your Postgres database with Power BI Desktop.
        4. Part 4 - Setup and configure the Power BI Gateway for cloud refresh.
        Part 1 - Provision your Azure Postgres Database and install the components for connecting to Postgres   In case you haven't provisioned an Azure Postgres database, here are the basic steps to creating a new instance of the Postgres database service via the Azure Portal.  
        1. Login to the Azure Portal, and select "Create a resource"
        2. Search the marketplace for "Azure Postgres", and select the top search result. (Note: there are several other marketplace templates for Postgres, but for the purposes of this blog I'm focusing on the Azure Database for PostgreSQL)
        3. Click "Create"
        4. Fill out the form for the new server including the Server name, Resource group, Server admin login name, password, and click "Create". (Note: there are other settings here like the Postgres version and the number of cores, but they aren't important to this scenario).
        5. While your database is being provisioned there are a couple of things you can go ahead and setup.
        6. Install Npgsql 3.1.8 via the MSI (note this is version specific), you can find the installer here: https://github.com/npgsql/npgsql/releases/tag/v3.1.8
        7. By default the postgres service has SSL enabled. You need to download and trust a specific certificate so SSL communication will work. (you could turn off SSL for the server in Azure, but that's cheating).
      Part 2 - Register the appropriate certificate so SSL connections to Postgres work.   There official version of these instructions can be found here: https://docs.microsoft.com/en-us/azure/postgresql/concepts-ssl-connection-security. These directions are intended to be generic and therefore cover many scenarios.   Since the directions are a bit terse and interleave linux directions with windows directions, I decided to give you an abbreviated step by step specific to Windows (Note: Power BI and the Gateway presently only run on Windows):
        1. Download the encrypted cert from: https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt.
        2. Download and install openssl to decrypt the cert https://slproweb.com/download/Win32OpenSSL_Light-1_1_0h.exe.
        3. Run the command: openssl x509 -inform DER -in BaltimoreCyberTrustRoot.crt -text -out root.crt (Note: depending on where you install openssl and/or where you dowloaded the .crt file to you may need to amend this command to deal with paths.)
        4. Double click on the created root.crt file, you'll be presented the certificate info. Click "Install Certificate…"
        5. Select "Local Machine", and click "Next"
        6. Select "Place all certificates in the following store", and "Browse"
        7. Select "Trusted Root Certification Authorities", and click "Ok"
        8. Click "Next".
        9. Click "Finish".
        Part 3 -   Connect to your Postgres database with Power BI Desktop   Now that we've installed all the prerequisites I'm going to walk through all the steps necessary to connect to the database.   Since this blog is largely about Power BI, I'm going to make the wild assumption that you already have Power BI Desktop installed. Once it's installed here are the steps:  
        1. Let's go to the azure portal for a bit to grab and configure things, login to the portal if you don't still have it open, and navigate to the Postgres resource you created
        2. Take note of the "Server name" and "Server admin login name", you'll need these for the gateway config.       Second, click on "Connection security", and add a firewall rule for the IP address of your machine running Power BI Desktop. The easiest way to do this is with "Add Client IP", then make sure you save your changes.
        3. Now launch Power BI Desktop.
        4. Use the "Get Data" menu to select a data source and select "More…"
        5. Select "Database" and "PostgreSQL Database" and click "Connect"
        6. Enter the server name you made note of in step 1, and the database name. (Note: for my purposes I'm just using the default database postgres as I'm only proving the scenario works).
        7. Enter your username and password, and click "Connect" (Note: the username needs to be in the format username@servername)
        8. In the data source navigator, select the tables you want to include in the model and click "Load".
        9. Check the "Fields" pane to ensure you've got some tables in your data model.
        10. You're done, save a local copy of the file for later publishing, or publish directly to Power BI using the "Publish" button.
          Part 4 - Setup and configure the Power BI Gateway for cloud refresh   At this point all you have a Power BI model connected to Postgres, but most companies I talk to actually want to publish their models, share them with others, and most importantly do some sort of a scheduled refresh on the data. In order to do this you're going to have to setup the Power BI Gateway somewhere.   This gateway can be installed on a Windows machine ranging from your personal desktop to a server. The most important things are that the machine should be on all the time, and it should have internet connectivity; therefore, installing it on your laptop is typically not a good idea. I chose to provision a Windows VM in my Azure subscription. Once you've decided where you're going to install the gateway here are the steps to get it running:
        1. Go through all the steps in "Part 2" again on the gateway host machine. (Pro Tip: You've already decrypted the certificate with OpenSSL on your Power BI Desktop machine, so you can bypass steps 2 & 3 by copying the root.cer file to the target machine).
        2. Download and install the Power BI Data Gateway from: https://powerbi.microsoft.com/en-us/gateway/ and click "DOWNLOAD GATEWAY"
        3. In the installer, you can accept all the defaults. In particular, make sure you select this option in the installer
        4. Once the install is complete you'll be taken through the configuration steps. First enter the email address you use to login to Power BI and click "Sign In"
        5. Next, select "Register New Gateway" (Note, the other option is useful if you need to migrate a gateway to another machine or are rebuilding a machine).
        6. Now you need to configure your gateway by giving it a unique name so you can find it in the Power BI Portal, and set your recovery password which is necessary if you ever need to move / restore your gateway.       (Note: for more advanced install scenarios you can add your gateway to a gateway cluster enabling high availability for the gateway).
        7. Gateway installation and configuration is complete, you should see a screen like:
        8. Now that the gateway is installed, we need to register the datasource with the gateway. This is done in the Power BI portal.
        9. Login to www.powerbi.com, and on your main screen select manage gateways, you'll find this under the settings gear.
        10. This will take you to the Add Datasource page, select the gateway corresponding to the name you gave the gateway in step 6.
        11. Click the ellipsis (…) and select "ADD DATA SOURCE"
        12. Name your datasource and pick the appropriate type.
        13. Just like we did with Power BI Desktop, lets jump over to the azure portal for a bit to grab and configure things. First, take note of the "Server name" and "Server admin login name", you'll need these for the gateway config. Second, click on "Connection security", and add a firewall rule for the IP address of your gateway server.
        14. Fill out the necessary server and database settings, for my purposes I just used the default database to prove it works.       (Note: Pay close attention to the fact that the admin login is username@servername), and click "Add"
        15. Note, if you get the following error its typically because you entered your password wrong, or you didn't add a firewall rule:
      Ok, you're all finished. Now when you decided to publish your Power BI Desktop file to the Power BI portal, you'll be able to manually refresh or setup a scheduled refresh on the data. At refresh time Power BI will use the gateway server to connect to the database and issue queries using the correct driver and feed the results back to the Power BI Service.]]>         Connecting to Blob Storage with a Shared Access Signature in Power BI https://docs.microsoft.com/archive/blogs/chmitch/connecting-to-blob-storage-with-a-shared-access-signature-in-power-bi Thu, 31 May 2018 11:57:38 GMT https://blogs.msdn.microsoft.com/chmitch/?p=165 As you're likely aware Power BI supports dozens of data sources out of the box enabling you to mash up data from a variety of sources.  Each one of these sources of data has a unique configuration governed by the capabilities of the source (ie. text files vs. a database).  At times you may also find that the capabilities of the connector can be governed by what features the developers of the connector chose to implement.  I ran into such a scenario recently using the Power BI connector for Blob Storage.  You may have noticed that out of the box the connector only supports two authentication methods.
    1. Anonymous - this will only work if you've made your blob container public.  (I find this is invalid for most scenarios).
    2. Account Key - This is the master account key, which has full permissions to the storage account.
      If you're familiar with Blob Storage, it has another option for authentication called "Shared Access Signatures".  This mechanism provides for much more granular security as it allows you to define at the container or blob level the type of access (Read/Write/Delete/List), and also when said access will expire.  (For more details see:  https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1)  Because of these added features it's a far more secure model to use SAS tokens as you can prevent users of the data from maliciously or accidentally abusing the data.  Unfortunately the Power BI Connector does not support the SAS token method of authentication. (Please upvote this feature request here: https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/9953427-connect-to-azure-tables-via-shared-access-signatur )   Fortunately Power BI can talk to all manner of unanticipated data sources that have an HTTP endpoint via the Web connector.  I find this particularly useful in two scenarios:
    1. Using an oData source that provides an authentication mechanism that is not exposed in the oData provider, most commonly oAuth2.
    2. Using a REST enabled endpoint.
      Azure Blob Storage happens to have a REST endpoint, so I set out on a mission to query Azure Blob Storage with a SAS token via the REST endpoint using Power BI's Web connector.  This was truly a mission, so I want to share this with you to reduce your heartache and encourage you again to upvote this feature in Power BI and share this with your friends.   In order to create this scenario I have to make some assumptions.  For our purposes here I'm assuming:
    1. You have a storage account and container created already.
    2. You want to process more than one file in the storage container (processing one file is actually just a dramatically simplified version of this).
    3. The storage container has some blobs in it.
    4. The blobs in the storage container all have the same format.
    5. The format of the files in the container is CSV (it should be pretty obvious where you'd change this logic in the process).
      Since there are several steps, I decided to break it up into three parts to better organize the process.  The three key steps are:
    1. Generate the shared access signature.
    2. Enumerate the files in your storage container in Power BI with the shared access signature.
    3. Manipulate the M script in Power BI to download the CSV files.
    Part 1 - Generate your shared access signature:
    1. First and foremost, you'll want to download Storage Explorer, it's by far the easiest way to generate a Shared Access Signature.  You can find Storage Explorer here:  https://azure.microsoft.com/en-us/features/storage-explorer/
    2. Once you've downloaded storage explorer connect to your storage account using the primary key, and navigate to a container in the storage account.
    3. Right click on the storage container, and click "Get Shared Access Signature…"
    4. Make sure "Read" and "List" are selected.  This will allow the same signature to be used to not only read items but enumerate items in the container.  (Note:  for security purposes I'm making my key very short lived.  In the real world you'll likely want to make the key have a longer lifetime).
    5. Copy and save the query string as we'll use it several times. The query string will look like: ?st=2018-05-30T14%3A50%3A00Z&se=2018-05-31T14%3A50%3A00Z&sp=rl&sv=2017-04-17&sr=c&sig=qb1hBqeASuJJRR1eZmPEQu1MLXTCLqn2G4bTlfDlcp4%3
      Part 2 - Connect to your storage container in Power BI Desktop.
    1. Launch Power BI and select "Get Data -> Web"
    2. Switch to the "Advanced" view, and input the base URL in the first box, your copied query string in the second box.
    3. Click "Add part" and enter "&restype=container&comp=list" in the third box (this is the directive that tells the rest endpoint to return a an enumeration of the blobs in the container.
    4. Click "Ok", you will eventually be presented with a screen that shows a one row table being returned.  This is because the payload that comes back from the REST api is XML.  Click "Load"
    5. This will load the table, which is not very useful, we'll need to manipulate it.  Click "Edit Queries"
    6. Delete all columns except for "Blobs" as they are unnecessary.
    7. Drill down into the Table twice to get at the underlying file names in the container.
    8. Now we have a list of filenames in the container, but we need to extract the individual files.  To do this we first need to build a complete URL for the file.  Click "Custom Column" from the "Add Column" menu.
    9. Build a formula that uses the URL to the container, the filename, and the querystring to create a complete URL, and name the field "FileURL".  For example:  "https://cgmwasb.blob.core.windows.net/ambulancedemo/" & [Name] & "?st=2018-05-30T14%3A50%3A00Z&se=2018-05-31T14%3A50%3A00Z&sp=rl&sv=2017-04-17&sr=c&sig=qb1hBqeASuJJRR1eZmPEQu1MLXTCLqn2G4bTlfDlcp4%3D"
      Part 3 - Manipulating the M script to download the blobs.
    1. Under the Home menu, select the "Advanced Editor" to see the M script.
    2. First, we need to add a comma to the end of the last line before the "in" keyword so we can add another expression.
    3. Second, we need to add another step to process the "Web.Contents" of each URL in our table.  To do this we can add a column to the table with the following M script.  The script will look like this: BinaryURLContents = Table.AddColumn(#"Added Custom", "BinaryFiles", each Web.Contents([FileURL]))
    4. Finally, we need to change the return object after the "in" statement from the script from #"Added Custom" to BinaryURLContents.
    5. The script should look like this:
    6. Now that we've told the M to explode each URL into a binary file, the rest of the steps can be done in the GUI.
    7. Remove all columns except the "BinaryFIles" column.
    8. Click the "Drill Down" button to expand the binary files.
    9. You'll be presented an error, because by default Power BI will try to process the files with the excel parser.  Click "Edit"
    10. Switch the parser to open the file as a "Csv Document" and click "OK"
    11. You'll now be presented the "Combine Files" dialogue which lets you adjust the delimiter and codepage if necessary.  Click "OK".
    12. You're done, you now have a merged set of all files in the container that were authenticated using a shared access signature.
      Phew...  I know that was a lot of steps.  Hopefully you'll find this useful and it will save you some time.  What's I find particularly nice about this approach is that it's defined entirely in the Power BI model, and will gracefully deal with a changing list of files in the container.  As new files show up in the container you don't need to update the model, a scheduled refresh will just read them as part of this process.]]>         Introducing the Azure SQL Data Warehouse DWU Calculator https://docs.microsoft.com/archive/blogs/chmitch/introducing-the-azure-sql-data-warehouse-dwu-calculator Fri, 16 Dec 2016 17:01:39 GMT https://blogs.msdn.microsoft.com/chmitch/?p=156 Azure SQL Database DTU Calculator this tool is designed to examine your existing workload and estimate the service tier required to fulfill the workload when migrated to Azure SQL Database.  I discussed this effort with Justin, and we decided it would be a great idea to follow the same pattern with Azure SQL Data Warehouse.  Like the DTU Calculator the this new DWU Calculator  is designed to be easy to use and give you a good idea where to start when provisioning your Data Warehouse.  Unlike the DTU Calculator it uses different metrics for its calculations to align better to usage patterns of data warehouses and the nuances of the Azure SQL Data Warehouse service. While I will get into more details on this utility, the premise of it is really quite simple.
    1. Run a utility via powershell or a simple executable to capture metrics from the server while under load.
    2. Upload the captured csv file to the calculator.
    3. Examine the recommended performance level(s) and determine your starting point. **Note:  This step is important because typically not all of the captured workload will fit in a single performance tier.  The calculator will tell you what percentage of the observed workload fits in each tier and make a balanced recommendation.
    I soft launched this utility (meaning I published it without telling anyone) back in July and haven't had any major issues.  Over the next few weeks I'll be cleaning up some documentation in advance of a more formal introduction.]]>         Sending events to IOT Hub over HTTP via REST https://docs.microsoft.com/archive/blogs/chmitch/sending-events-to-iot-hub-over-http-via-rest Wed, 16 Nov 2016 14:09:15 GMT https://blogs.msdn.microsoft.com/chmitch/?p=135
  • Use SAS tokens - While each device in the device registry has a primary and secondary key, these keys cannot be used for authentication on the rest endpoint. Only a SAS token can be used.
  • The SAS token must be URL encoded - There are some special characters in the SAS token ("/" and "=" are the ones I noticed). To make sure these are handled appropriately they must be replaced with the URL encoded values.
  • The SAS tokens must have a time to live - At first I was trying to be clever and use 0 for my TTL, thinking this would be an open ended SAS token.
  • The SAS token must be included in a specific HTTP header. - I couldn't find this documented anywhere, and in fact had to use fiddler to examine the packets and see what the header looked like.
    • Here's the pieces to make all this work:
    1. First we need to define the rest endpoint. The device communication endpoint for sending messages to the IOT Hub is at /devices/{deviceid}/messages/events. To address this in my code I used the following:
      string iotHub = "[iothub]"; string deviceId = "[deviceid]"; string api = "2016-02-03"; //Build rest endpoint for the device. string restUri = String.Format("https://{0}.azure-devices.net/devices/{1}/messages/events?api-version={2}", iotHub, deviceId, api);
    2. Next we need to generate a SAS token for the IOT device. This can be done programmatically, or using the handy Device Explorer tool included in the SDK to create a device and generate a token.   (https://github.com/Azure/azure-iot-sdks/tree/master/tools/DeviceExplorer). There are two things to keep in mind with this tool: You must enter a valid TTL (> 0): p1The output contains more than just the SAS token, select the portion of the SAS token after "SharedAccessSignature=", this is the URL encoded SAS token: p2
    3. In order to package the header appropriately and send the message I used the following code. Here we add the generated sas token and add it to the "Authorization" header.   Note the "body" is just any class that is JSON serializable.
      string sas = "SharedAccessSignature sr=cgmIOTTelemetry.azure-devices.net%2Fdevices%2FmyFirstDevice&sig=hFvm0tDU4ytu85efPpE3T7VYhFUNII4NigdonRJjKoA%3D&se=1479398100"; HttpClient client = new HttpClient(); client.DefaultRequestHeaders.Add("Authorization", sas); var content = new StringContent(JsonConvert.SerializeObject(body), Encoding.UTF8, "application/json"); var result = client.PostAsync(restUri, content).Result;
    Once all this is complete, you have a very basic authenticated call to the IOT Hub's REST endpoint. What's nice about this is unlike event hub, each device is authenticated. Therefore, if you have a problem with a specific endpoint you can disable it and/or regenrate it's keys and tokens without taking all your other devices offline. I wrote this code in C# because I'm most efficient in that language, but I hope you see that all we're doing is adding a specific authorization header and sending a json payload to a URL via an HTTP Post. Any language that can do an HTTP post, and add a string to the header can do this same logic. As usual I've included this code in my github at: https://github.com/chmitch/azure-iot-send-via-rest]]>         Long Term Backup with Azure SQL Database https://docs.microsoft.com/archive/blogs/chmitch/long-term-backup-with-azure-sql-database Tue, 08 Nov 2016 14:14:28 GMT https://blogs.msdn.microsoft.com/chmitch/?p=65 Extended Backup Retention With Azure SQL Database", I outlined a method for extending the boundaries of the Azure SQL Database service. To refresh your memory the defaults here are 7 days for the Basic tier and 35 days for the Standard and Premium tier of Azure SQL Databases. While the method I outlined in this earlier blog post had some limits and required some automation via PowerShell, it was really the only method we had to overcome this obstacle. I'm pleased to report that the SQL Azure team heard the demand for configurable backup loud and clear, and has provided a feature to extend these limits to 10 years! For the official announcement you can take a look at the Azure Blog, but for the sake of completeness I thought I would give a step by step guide to elaborate on a few more details that aren't spelled out in the official posting.
    1. While we logically think about database policy at a "database" level, you actually control this feature at a "server" level. So avoid your instinct to navigate to the SQL Azure Database and go to the server instead. You'll find the tab to enable this capability under "Settings". ltb_1
    2. Since this feature is in preview, you must accept the preview terms before it can be used. ltb_4
    3. As we earlier mentioned this policy applies at the server level; therefore, the next step is to select all the databases on the server you want it to apply to. ltb_3
    4. Select the Recovery Vault you'd like to use. ltb_4
    5. Finally define your retention policy and you're ready to go. ltb_5
    A couple things to call out that were a bit of a surprise to me:
    1. Presently this long term retention imposes the restrictions that the vault must be in the same subscription, region, and resource group. If you don't have this you'll see a warning as follows: ltb_6
    2. While this give you a far longer duration of retention than previously available, there is not an ability to do any fine grained policies (ie. keep daily backups for 1 year, and weekly backups for 3 years, etc.)
    ]]>         Automating database migration to SQL Azure https://docs.microsoft.com/archive/blogs/chmitch/automating-database-migration-to-sql-azure Mon, 03 Oct 2016 12:16:13 GMT https://blogs.msdn.microsoft.com/chmitch/?p=35 Migrating databases to a new environment can be a tedious but necessary task.  For years DBAs have been dealing with the process of rolling out SQL Server upgrades, and have achieved this in On Premises environments in a variety of ways including: 1.       Backup and Restore database 2.       Attach / Detach database 3.       SAN magic (typically snapshotting LUNs or moving LUNs from one host to another) 4.       In Place upgrades (Note:  just because you can, doesn’t mean you should) Over the years a variety of tools have been created to support the tedium of these migrations including Scalability Expert’s SQL Server Upgrade Assistant which aids in detecting compatibility issues with your database upgrade.  My intent in writing this blog post is to focus on brute force automation of the migration process.  The topics of “what tools are available to assess the compatibility of my database” and “what features in SQL Server are not supported Azure SQL” are rather well documented.  For the sake of completeness here are two resources to help you out in that regard: ·        Carl Rabeler, formerly of the SQL Customer Advisory Team, has a great article (SQL Server database migration to SQL Database in the cloud) that gives a detailed audit of all the tools available for taking a database from an on Premises Database to the cloud and goes into their pros and cons of each tool. ·        There is also an article by Rick Byham (Azure SQL Database Transact-SQL differences) that gives a good overview of specific transact SQL features that are not supported in the cloud. With that backdrop let’s assume you have gone through the process of assessing your databases and determined the databases are ready for the cloud.  Now you are ready to execute the migration, but are looking at the volume of work and decided “right click -> migrate database to sql azure” is too labor intensive, what’s next?  The answer… automation, to do this let’s take a brief look at what the migrate wizard does. pic1 Notice, this is a one step process according to the wizard, and unfortunately it doesn’t give you an opportunity to script things out either.  To run, it effectively asks for just a few pieces of information: 1.       Source Database which is defined by which database you right click on. 2.       Destination Details: a.       What Azure SQL Server do you want this database on? b.       What should the database be called? c.       What size is the database? 3.       Where do I store the intermediate bacpac file? To script the same capabilities, we need to use the following: ·        The SqlPackage utility, which is a command line tool that can be used to generate bacpac and dacpac files from your database among other things. ·        The New-AzureRMSqlDabaseImport command to restore the generated bacpac file as an Azure SQL Database. The following commands are snippets that conceptually handle each part of the process. Part 1, execute SqlPackage to save the file locally:
    #setup the local variables. $server = "localhost" $database = "Database1" $path = "c:\projects\" + $database + ".bacpac"   #Define the path to the sqlpackage.exe and the command line parameters $cmd = "C:\Program Files (x86)\Microsoft SQL Server\130\DAC\bin\sqlpackage.exe" $params = "/Action:Export /ssn:" + $server + " /sdn:" + $database + " /tf:" + $path $p = $params.Split(" ") #shell out the export command for the bacpac & "$cmd" $p
    Part 2, upload the generated file to Azure blob storage:
    #set variables $fileName = "Database1.bacpac" $storageName = "[storage account]" $storageKeyType = "StorageAccessKey" $storageContainer = "backups" $storageUri = "https://$StorageName.blob.core.windows.net/$storageContainer/$filename" $localpath = "c:\Projects\$filename" $StorageKey = "[storage key]"   #Set storage context and upload blob file $context = New-AzureStorageContext -StorageAccountName $storageName -StorageAccountKey $storageKey Set-AzureStorageBlobContent -File $localpath -Container "backup" -Context $context
    Part 3, restore the uploaded bacpac file as a new database:
    $importRequest = New-AzureRmSqlDatabaseImport   –ResourceGroupName $ResourceGroupName ` –ServerName $ServerName ` –DatabaseName $DatabaseName ` –StorageKeytype $StorageKeyType ` –StorageKey $StorageKey ` -StorageUri $StorageUri ` –AdministratorLogin $credential.UserName ` –AdministratorLoginPassword $credential.Password ` –Edition Premium ` –ServiceObjectiveName p1 ` -DatabaseMaxSizeBytes 50000
    I pull all of these things together in a more sophisticated script that accounts for scenarios like choosing the service level of the database based on its size, and uses SQL Power Shell to enumerate the local databases.  Finally the script uploads the content and restores the database in SQL Azure.  You can find the complete script on my git hub page .]]>         Extended “Backup” Retention with Azure SQL Database https://docs.microsoft.com/archive/blogs/chmitch/extended-backup-retention-with-azure-sql-database Fri, 23 Sep 2016 09:39:26 GMT https://blogs.msdn.microsoft.com/chmitch/?p=25
  • Premium (P1 – P15) and Standard (S0 – S3) Databases – 35 days of Point in Time Restore
  • Basic Database (B) – 7 days of Point in Time Restore
    • A question I’m often asked is whether this feature is configurable on a per database basis.  Most customers I speak with would gladly pay the extra storage costs to make this a tuneable feature on the database, and add the option of sacrificing the Point in Time nature of the restore as the backups age out.  Another question I’m asked is how to best take a preemptive backup before a deployment so there is a known good state to recover to in the event of a deployment failure.   The built in backup can aid in the deployment scenario just fine, but the challenge is often understanding what “point in time” to recover to.  Those responsible for a deployment rest easier knowing there is a snapshot of the database when all else fails. While these features (tuneable retention & on demand backup) aren’t addressed as part of the built in backup capabilities of Azure SQL Database there is a way we can use other features of the database in order to achieve this namely SQL Database Copy and SQL Database Export.  While not technically “backup” these features can be used to get you a consistent copy of your database for disaster purposes, hence the quotes.  (NOTE: When I put the quotes there I have a mental image of Mike Myers aka Dr. Evil doing air quotes.) Let’s explore these two features a bit… SQL Database Copy – is designed to make a transactionally consistent copy of your database leveraging the built in backup retention policy.  It first makes a copy of the tail of the transaction log, and then uses this tail plus the full, differential, and log backups retained behind the scenes in Azure SQL Database.  All this is done without interrupting the log chain of the current database retention plan (a common problem with preemptive full backups in on premises databases). SQL Database Export – is designed to export a copy of your database to a .bacpac file which is an encapsulated version of the databases schema and data.  While not truly a backup file like a .bak file in a traditional SQL database, it does contain everything you need to reconstitute your database in the event of an emergency. Based on these two descriptions, you might ask yourself “why do I need both?”.  This is the point where the fine print is important.  There are two reasons I think it’s essential to use both of these features in concert:
    1. SQL Database Export is not transactionally consistent. Therefore, it is strongly recommended that you have a database that is not being actively used while you take the copy.  Hence I would recommend starting with database copy, then export, and ultimately drop the copied database.
    2. SQL Database Copy creates a copy of your database. Therefore, if you have a 50 GB P1 database after the operation you’ll have two 50GB P1 databases, and consequently double the cost of hosting the database.  Each time you run this operation you’re increasing your cost, and each database gets its own retention schedule which is unnecessary.  Using the bacpac approach you keep the database copy around for a small amount of time and keep your costs low.  You also get a choice whether you want this retention in Azure Storage which is much cheaper than Azure SQL to use or to take the bacpac out of Azure and into your own data center.
    Now that we have a good background on why I’m recommending this approach, here are a few powershell script segments to get you started.  I logically break this down into three discrete steps:
    1. Initialize variables.
      # Define the source datbase, server, and resoruce group $sourceDatabase = "[source database]" $sourceServer = "[source server]" $sourceResourceGroup = "[source resource group]
      # Define output location for bacpac $storageAccount = "[blob account].blob.core.windows.net" $storageContainer = "[blob container]" $storageKeyType = "StorageAccessKey" $storageKey = "[storage key]"
      # Admin credentials for the database $adminUser = "[admin user name]" $adminPassword = "[admin password]"
      # Generating a unique name for the databse copy and the uri for the bacpac. $destinationDatabase = $sourceDatabase + (Get-Date).ToString("yyyyMMddHHmm") $bacpacUri = "https://" + $storageAccount + "/" + $storageContainer + "/" + $destinationDatabase + ".bacpac"
    2. Create the database copy. # Execute the copy command New-AzureRmSqlDatabaseCopy -ResourceGroupName $sourceResourceGroup ` -ServerName $sourceServer ` -DatabaseName $sourceDatabase ` -CopyDatabaseName $destinationDatabase
    3. Export the bacpac to the storage account. # Note:  Exporting to a bacpac requires you to have an account and password for an admin. #        If you do not include this the command will ask for it at execution time. $exportRequest = New-AzureRmSqlDatabaseExport   –ResourceGroupName $sourceResourceGroup ` –ServerName $sourceServer ` –DatabaseName $destinationDatabase ` –StorageKey $storageKey ` -StorageUri $BacpacUri ` –StorageKeytype $storageKeyType ` –AdministratorLogin $adminUser ` –AdministratorLoginPassword $adminPassword
    4. Cleanup the database copy. #Now that the export is done, delete the databse. Remove-AzureRmSqlDatabase  -ResourceGroupName $sourceResourceGroup ` -ServerName $sourceServer ` -DatabaseName $destinationDatabase
    This is the gist of it.  In order to save yourself some headaches with copy and paste, and for a complete version of the powershell script checkout my github repository @ For a complete script you can check out my github repository ]]>