Patrick’s Azure Blog

No data for you!

Patrick El-Azem — Mon, 28 Aug 2017 10:35:40 GMT

Has this happened to you?

- "We need a copy of production data" - "Why?" - "For {staging | testing | baselining | repro-ing | exploring big data on Azure | doing my job }" - "No."

Getting data can be done by taking a snapshot or copy of production data - like a transactional database backup or copying a set of data files - and bringing this data to the needed environment. But production data may contain sensitive information, such as Personally Identifiable Information (PII) or Protected Health Information (PHI). To protect customer privacy and comply with applicable regulations and compliance frameworks, such data is usually stored in secured environments, and subject to access and movement controls. Such controls may prohibit providing the data - even to internal teams that need "real-world" data. ... Which means the answer to "We need production data" may be a completely justifiable - but very inconvenient - "No".

But we need that data. Now what?

We need data to move our project ahead. What are some strategies to consider?

Use public data

In some cases, data that is structured identically to production data is not needed. Many public data sets (like these and these) are available for development, testing, and training. The downside, of course, is that public data sets come with a pre-existing structure, so they will not be useful if your work requires your existing data structure, or any control over the data structure.

Scrub or mask

To mitigate concerns about inappropriate disclosure of sensitive data, "scrubbing" or "masking" approaches can obscure, alter, or remove sensitive data from copies or snapshots of the data (for example, see my post on Dynamic Data Masking). The specific approaches vary depending on the nature and sensitivity of the data, ranging from manual, to automated, to using tools like Delphix. In general, an understanding of the data - especially which elements are sensitive - is needed, though more sophisticated approaches or tools can infer what parts of the data need protection by looking for patterns like email address, social security or insurance number, and so on. However, in highly regulated or very risk-averse environments, scrubbing or masking approaches may not be acceptable due to the concern of "what if we miss something, and inappropriate disclosure happens despite our scrubbing or masking measures?". Something as simple as finding (and scrubbing or masking) permutations of personal identifiers in large text fields, for example, can be challenging to automate with very high accuracy across many records.

Synthesize

When pre-defined data sets are not suitable, and when scrubbing or masking is not appropriate, we can consider synthetic data. This means artificially generating data that resemble our production data, and can be substituted for it. In this way, there is no risk of inappropriately disclosing sensitive data, since production data is not accessed or copied. Initially, we need to mimic our production data structure. This is straightforward, as long as production file or database schemas (i.e. structures) are available. Next, we need to think about how to generate the synthetic data. The right approach to generating synthetic data that is "close enough" to production data could start simple - for example, duplicate the same record thousands of times, perhaps with an auto-incrementing identifier field. However, we will probably want to adopt a more nuanced approach to data synthesis that emulates, on a field-by-field basis, the characteristics of our production data as closely as possible. For numeric fields, this may mean generating values in a set that has similar statistics to the production data: for example, a similar mean and standard deviation for normally distributed data, the same min and max for uniformly distributed data, and so on. We may have fields constrained to one of a specific list of allowable values - in data science, known as a categorical feature or variable (for example, days of the week, or colors of the rainbow, or other closed sets). We may even want to weight individual values differently, so that one value appears more often than another. We may also have fields that fall within a specified range - such as dates that fall within a start and an end date - or fields that we want to generate dynamically, perhaps via some sort of code fragment or anonymous function. The more dynamic, real-world, and closer to production data we want our synthetic data to be, the more effort we will need to invest in analyzing our production data's characteristics and finding ways to emulate these characteristics at scale. There are many ways of doing this; the details will depend on our environment and controls. (Azure Machine Learning and Azure Data Catalog provide excellent data summary statistics.)

Generator Tool

I wrote a file data generator tool for a recent project where a new environment was being architected in Azure. Validation of this environment required system and load testing with real-world (volume and variety) data. Production data was highly sensitive, there was no non-production data available, and implementing scrubbing/masking was not realistic in our timeframe. The synthetic data generated with this tool enabled environment validation - and the project - to continue. The rest of this post discusses the tool I wrote, what it does, how it does it, how you can use it, and what else is planned for it. I called it the Synthetic Data File Generator (or sdfg below) because sdfg is easy to type and because I don't have an MBA in catchy product naming.

How can you use sdfg?

First, let's get off to a quick start. We'll get sdfg onto your machine and ensure it runs. Then, we'll dive deeper into a technical discussion.

Quick Start

You will need a machine running Windows 10 x64, or Windows Server 2016 x64. (Why? See Technical Details, below.)
Go to https://github.com/pelazem/syndatafilegen/tree/master/run and download `sdfg-win10-x64.zip`.
Extract the zip file contents on your machine.
Open a command prompt in the folder where you extracted the zip file.
Start by typing just `sdfg`. You'll see a prompt that you need to specify the path to a runfile.
What is a runfile? Briefly - a runfile contains your instructions that tell sdfg what synthetic data files to generate. See below for a detailed discussion. The zip file you downloaded includes a `/runfiles` folder that contains several example runfiles; these are also available at https://github.com/pelazem/syndatafilegen/tree/master/runfiles. We'll start by using one of those.
I used the `csv.json` runfile, which instructs sdfg to generate comma-separated values (.csv), at hourly increments over 30 days, with 100-200 records per file. As you can see, that took a little over three seconds.
File Explorer shows that sdfg generated 744 .csv files. These are small files and a small amount of data, but if you have gotten this far, you have run sdfg and our Quick Start is successful. Optionally, you can run sdfg with the other sample runfiles in the .zip you downloaded. Check the `/runfiles` folder; at the time of this writing, in addition to `csv.json`, sdfg also includes `arff.json` (ARFF is a file format used by the Weka ML tool and supported in Azure ML) and `fixedwidth.json` generic runfiles.
A quick look inside one of the generated .csv files shows, yes, comma-separated data.

Runfiles

Now that we have generated a first set of data files , I'll discuss runfiles: what they are, how they are structured, and how you can create your own for your specific file structures.
Runfiles are JSON files that describe the synthetic data files to be generated. A runfile contains all the information sdfg needs. (My goal was to enable a zero-code experience, and require only a text editor to customize sdfg output.)

sdfg runfiles include the following elements:

Generator: instructions for sdfg that are separate from the file structure specifics. Examples include where to write the output, and what date range to use for generated files.
FileSpecs: one or more top-level elements, each of which contains all file-specific details as well as child elements for the fields (e.g. columns in a .csv file) to be written.
FieldSpecs: within each FileSpec, one FieldSpec element for each field to be written.

Let's examine runfiles in more detail.
For reference, please consult /runfiles/TEMPLATE_FULL_RUNFILE.json, which is available both in the .zip download as well as at https://github.com/pelazem/syndatafilegen/tree/master/runfiles. This runfile is a template that includes all supported keys and values, with explanatory comment for each.
Thus, while I'll discuss several important elements below, please consult this template for details as you build your own runfile. In particular, some elements are constrained to a set of allowable values (for example, which file type you need, such as CSV or fixed-width), and the template runfile documents those.

Generator

Generator is a top-level element. Generator contains OutputFolderRoot and DateStart/DateEnd child elements. OutputFolderRoot is a relative or absolute path on the local machine's file system. This is the root folder under which generated files will be written by sdfg. The dates are used for two purposes: first, if output file paths include date/time tokens (see below), dates from within this range are substituted for the tokens and second, if so specified, dates from within this range are written to the output files as a "when did this record occur" event date/time tracking field.

FileSpecs

Runfiles include one or more elements under FileSpecs. This means that a runfile can include instructions to generate multiple file sets. For simplicity, keep it to one FileSpec element at first. I'll discuss several FileSpec child elements here; others are simple and self-explanatory. See /runfiles/TEMPLATE_FULL_RUNFILE.json for full details. FileType is the output file type. Currently, these are supported: arff, avro, delimited, fixedwidth, and json. RecordsPerFileMin and RecordsPerFileMax specify a range for how many records should be written to each generated file. The exact number of records will be randomly generated for each file from within this range. (After all, in the "real world", data files don't all have the same exact number of records either.) Need to specify an exact number anyway? Just set both values to your exact record count. PathSpec is how you tell sdfg what folder and file names to generate. You can specify an explicit path here (such as "output/myfile.csv"), or you can provide a tokenized value. This tells sdfg to iterate from Generator/DateStart to Generator/DateEnd, and to replace appropriate parts of your PathSpec with the corresponding date/time as it iterates through your date range. Let's illustrate with a couple of examples.

"PathSpec": "{yyyy}\\{MM}\\{dd}\\{hh}{mm}{ss}.txt" will generate output files under a year folder, then a month folder, then a day folder, then a file named for the hour, minute, and second, with a .txt extension. For example: 2017\\07\\18\\145325.txt
"PathSpec": "{yyyy}-{MM}-{dd}-{hh}.arff" will generate output files with a filename of year-month-day-hour and a .arff extension. For example: 2017-07-18-14.arff
"PathSpec": "{yyyy}\\{MM}\\{yyyy}-{MM}-{dd}-{hh}.csv" will generate output files under a year folder, then a month folder, then a file named for the year, month, day, and hour. For example: 2017\\07\\2017-07-18-14.csv

Notes on PathSpec: as you see, you can use a token (like {yyyy} in the last example) more than once, in both folder and/or file name specification. Also, note the double backslashes (\\): Windows paths use a backslash, but backslash is also the escape character in JavaScript so in a JSON file, I have to use \\ to mean \. FieldNameForLoopDateTime is the field name to which you would like an event date/time written. This is optional. If you specify it, then as sdfg iterates from Generator/DateStart to Generator/DateEnd, as each record is generated, the "current" date/time within this range is written to it. This is a great way to simulate event log files, which have a strictly increasing event occurrence date/time field on each record. As sdfg iterates through Generator/DateStart - Generator/DateEnd, the incremented loop date/time is adjusted at each iteration with a random coefficient so that successive records are not always incremented by the identical time interval. In other words, a little bit of event interval "jitter" or irregularity is introduced. Delimiter and Encloser specify strings used to separate (delimit) and enclose, respectively, values in each generated record. These are used only in delimited or fixed-width files. They can be one or more characters long. Using "Encloser": "\"" (note that since this is JSON, I have to escape the double quote symbol, hence the backslash) specifies that each value should be enclosed in double quotes; this is very useful, for example, when generating .csv files with numeric fields that are formatted with a thousands separator which, in some languages/cultures, is a comma. Enclosing values with commas in double quotes prevents downstream consumers of these files from mistakenly interpreting a thousands separator as a delimiter between fields. Thus, in this fragment of a .csv record "1000000","2,461","$14,082.50","Wednes..., there are two numeric fields present. The first is 2,461; the enclosing double quotes ensure that this value does not get split into 2 and 461 downstream. Similarly, the enclosing double quotes ensure that $14,082.50 is not split into $14 and 082.50 downstream. Three elements are available for fixed-width files. The first, FixedWidthPaddingChar, specifies what character to use to pad (i.e. fill to the specified width) a field in a fixed-width file where the value is shorter than the field length. For example, if I have a field with a width of 20 characters and I generate a value of "123" for it, sdfg will add 17 of the character specified for FixedWidthPaddingChar so that exactly 20 characters will be written to the field. The value specified for FixedWidthPaddingChar must be exactly one character long. Typically, this is a space. FixedWidthAddPadding and FixedWidthTruncate specify where to pad too-short values, and where to cut off too-long values, respectively. Both are constrained to either of two allowable values: "AtStart" or "AtEnd". This lets you control whether too-short values (like in the previous example) are padded out to the field width before or after the value, and whether too-long values are truncated (cut off) at the right or at the left. Each FileSpec also contains a list of FieldSpec elements, which are discussed next.

FieldSpecs

Each FileSpec element contains a list of FieldSpec child elements. Each FieldSpec element describes how that field should be generated. (In many, but not all, file formats a field is just a column. Because not all file formats are columnar, though, I use the term field rather than column throughout.) As with FileSpec above, I'll discuss the most important FieldSpec configuration elements below. Again, see /runfiles/TEMPLATE_FULL_RUNFILE.json for full details. FieldType designates the type of data that sdfg will generate for this field. Currently, this is limited to one of the following allowable values: Categorical, ContinuousDateTime, ContinuousNumeric, Dynamic, and Idempotent. Let's examine each of these.

Categorical designates a field whose generated values are randomly selected from a list of allowable values which you specify separately (see Categories below).
ContinuousDateTime designates a field whose value will be randomly generated from a min/max range which you specify separately for the field (see DateStart and DateEnd child elements of FieldSpec).
ContinuousNumeric designates a field whose value will be a number randomly generated based on the additional configuration you specify, including NumericDistribution and MaxDigitsAfterDecimalPoint.
Dynamic designates a field whose value is generated from a C# anonymous function you specify (compiled at runtime using .NET Roslyn) separately in the DynamicFunc element (see below).
Idempotent is used in advanced scenarios when you pass in already-existing data programmatically, using sdfg's file format serialization capability without its data-generating capability. See below in this post for discussion of interacting with sdfg's engine programmatically.

EnforceUniqueValues specifies whether sdfg should ensure that, per file, the values generated for the field are unique, i.e. that no values are repeated in this field throughout the file. By default, this is set to false. Setting it to true increases sdfg's memory usage and compute time, so ensure you really need it before setting it to true. FormatString is an optional .NET format string. If specified, it will be used to format the generated output. For example: "FormatString": "{0:c}" formats the generated value as a currency; "FormatString": "{0:n0}" formats the generated value as a number with zero digits after the decimal point; and so on. See https://docs.microsoft.com/en-us/dotnet/standard/base-types/formatting-types for details on .NET formatting strings. PercentChanceEmpty optionally enables sparse data. The default is zero. If set to a value greater than zero, then as each record is generated, in the field where this is specified, there will be that percent chance that the generated value will be empty. This is to enable generation of files that, like the "real world", may have missing data in some records. I implemented this as a probability per record. So, if you specify 15 here (for 15% chance a given record's value will be "empty"), the final output file will have a percent of empty values in this field that is close to 15, but not necessarily exactly so. This also yields (hopefully) more realistic files where, even if you specify the same value for PercentChanceEmpty on two fields, the generated records will have some records where neither field is empty, some where one but not both is empty, and some where both fields are empty. In other words, I tried to avoid artificially neat generated data. Since this is specified per field, generated files can be very finely tuned to resemble real data. For example, if production files always have an event date/time and a record ID, but sometimes have empty values in transaction or event fields, this setting enables simulating such a scenario on the applicable fields that sometimes are missing data. Similarly, EmptyValue enables specification of a string value to be written to the output when the value generated for a record in this field comes up as "empty". By default this is an empty string, but can be set to other values as needed. Again, this is set per field for maximum control over the output. FixedWidthLength specifies the size for the field in a fixed-width file. It is ignored in other file types. FixedWidthPaddingChar, FixedWidthAddPadding, and FixedWidthTruncate are all optional per field. If specified, they will override the same-named file-level settings. This enables you to optionally format each field in a fixed-width file individually if the file-level defaults are not applicable across all fields in the file. These are ignored for file types other than fixed-width.

Categories is a list of allowable values for a Categorical field. Optionally, values can be weighted so that some values will appear more often than others in the generated files. Weights are normalized by sdfg during processing, so you can specify any positive values that you like.
In the screenshot, I provide days of the week as my allowable values. I have weighted the days differently; for example, Friday is weighted 1.0 and Saturday is weighted 0.25, which means Friday is four times as likely to occur in the output as Saturday.

NumericDistribution designates the distribution, and associated parameters, to be used to generate random values for a ContinuousNumeric field. Currently, the DistributionName is limited to one of these allowable values: Beta, Cauchy, ChiSquare, Exponential, Gamma, Incrementing, InverseGamma, Laplace, LogNormal, Normal, StudentT, Uniform, and Weibull. Each of the DistributionName values require associated parameters to configure the sdfg random number generator correctly. For example, if you specify "DistributionName": "Normal", you will also need to specify values for Mean and StandardDeviation. Similarly, if you specify "DistributionName": "Uniform", you will also need to specify values for Min and Max. As before, consult /runfiles/TEMPLATE_FULL_RUNFILE.json for details. The numeric distribution element and its child elements are exhaustively documented there, with the parameters required for each distribution clearly designated. DynamicFunc is where you provide a C# anonymous function for a Dynamic field. This function will be used to generate a value for this field on each record. Specifically, you need to provide a function that takes no parameters and emits an object. For example: "DynamicFunc": "() => System.Guid.NewGuid()" generates a new GUID (globally unique identifier) on each row. What could you do here? You could call a REST API, for example. Note that this function is called for every record, so that a computationally intensive (read: slow) function will significantly increase the time sdfg takes to generate your files.

Conclusion

This semi-exhaustive look at the internals of sdfg runfiles will hopefully help you build your own runfiles and generate synthetic data files for your specific scenario.
Next, we'll discuss how you can use sdfg without runfiles.

Advanced Scenarios

Using sdfg with a runfile, as discussed above, is convenient in that it only requires you to edit a JSON runfile in a text editor. This approach works very well to generate standalone files. But what if we need to generate sets of related, differently structured files? What if one file, for example, needs to use categorical values (i.e. the set of allowable values) which were generated for another file in the set? In this scenario, using sdfg with a runfile will not work, as the runfile approach generates one type of file at a time. (In future, sdfg may be enhanced to support runfiles that lay out multiple related files, and support the ability to link a Categorical field's Categories list to the output of another FieldSpec. If I get feedback that this is needed at the runfile level - i.e. without needing to dive into code - I will prioritize that feature. In the meantime, let's look at how to do this programmatically with sdfg.) To continue in this section, it will be very helpful if you've cloned the sdfg github repo (https://github.com/pelazem/syndatafilegen) to your machine. You'll need Visual Studio 2017 (any edition) with the latest update (currently release 15.3) and the .NET Core 2.0 SDK installed.

Opening the solution in Visual Studio, we see the sdfg main projects: SynDataFileGen.App and SynDataFileGen.Lib.

The CCLF17 solution node contains a separate application that programmatically uses the sdfg engine to generate file sets relevant in US health care. Specifically, this application generates file sets in accordance with the Centers for Medicare & Medicaid Services (CMS) Accountable Care Organization - Operational System (ACO-OS) Claim and Claim Line Feed (CCLF) specification for 2017, version 18.0. This file set, provided to Accountable Care Organizations (ACOs), typically contains extensive PII and PHI data. ACOs and other organizations processing these files need to accommodate large files containing sensitive data, whose structure can change over time. Load-testing a new processing environment that is not yet security-hardened and compliant is impossible with the real files due to their sensitive content; that was the motivation for writing the CCLF17 application. The following screenshot shows an example method (in CCLF17.Lib/CCLFGenerator.cs) which uses sdfg entities (IFieldSpec, FileSpecFixedWidth, Generator, and WriterLocalFile) to first prepare the appropriate field specifications (in a separate helper method), then instantiate a file spec for generating fixed-width files, then instantiating the sdfg Generator with values for the configuration items we discussed above (e.g. output folder, start and end dates) and running it. Note that the generator's results - i.e. the records generated for this file - are returned from this method. That's because this CCLF17 application implements the scenario I discussed above, where one file may have Categorical fields whose allowable values are generated for another file. By returning the generated records here, I am enabling code that coordinates between the different files and ensures that one file's generated values serve as the categorical, allowable values for another file. The next screenshot shows code that calls the method in the previous screenshot, stores its results in an overall list (CCLF1S), then extracts a list of categorical values (CUR_CLM_UNIQ_ID). And finally, the next screenshot shows the categorical value list we extracted in the previous call (and previous screenshot) being passed into the next file generation method, where it will be used as the allowable value list for a categorical field. In this way, we have dynamically generated a list of values for one set of files, then we use those dynamically generated values to constrain related fields in other files. I won't cover more of the CCLF17 codebase here. Its overall structure is simple; there is extensive code to implement the structural details of the source file set, but it's fairly repetitive and if you are familiar with Visual Studio and .NET development, you'll have no trouble understanding it. Feel free to examine it and contact me with questions or comments. In summary, the github repo contains a solution that includes both the core sdfg utility - which outputs the executable we invoked in the Quick Start above - as well as a separate application that interacts with sdfg programmatically to implement a complex scenario that is (currently) not feasible with a JSON runfile.

Performance

sdfg's performance will vary based on the specs of the machine you run it on, and how complex you make your runfile (especially that DynamicFunc!). For a point of reference, I used the sample csv.json runfile for three separate sdfg runs. In each run, I only changed the FileSpec/RecordsPerFileMin and FileSpec/RecordsPerFileMax values; for each run, I multiplied them by a factor of 100X from the previous run. As you can see in the following screenshot, I used ranges of 100-200; 10,000-20,000; and 1,000,000-2,000,000 records per file. The sample csv.json runfile generates 744 files - one file per hour for 31 days. Let's examine a few of the results. On run 2, generating over 11 million total rows took less than 2 minutes; and while run 2's min and max records were scaled 100X compared to run 1, total time to execute scaled by 31X. On run 3, I generated over 1.1 billion records in less than 2.5 hours; the generated files added up to 115 GB. Remember: this is with randomly generated data (including a DynamicFunc) on every single record. Technical details: these tests were run on a Windows 10 Pro x64 machine, Creators Update (1703), with an Intel Core i7-6700 CPU, 32GB of RAM, and Samsung 850 Evo SATAIII SSD. The machine was otherwise loaded (i.e. not quiesced for max performance), running antimalware and other applications. This was a Release build of sdfg.

Technical Details

I wrote sdfg (and the CCLF17 application) as a .NET Core application and a .NET Standard library. Currently, sdfg is at .NET Core/Standard 2.0. The zip file you downloaded in the Quick Start is named sdfg-win10-x64.zip, and I said you need a Windows 10 x64 or Windows Server 2016 x64 machine to run it. Why the OS and x64 architecture constraint? Because .NET Core gives us the ability to ship the runtime with the application! In the Quick Start, I didn't say your machine needed .NET vAnything, right? That's because the zip file contains all the .NET it needs. Your machine doesn't need to have it installed just to run sdfg. This is a change from the .NET Framework (currently 4.7), which did require a system-wide installation. Distributing an application built for .NET 4.7 required you to have that .NET framework installed on that machine. By contrast, distributing an application like sdfg as a self-contained application means you can deploy multiple applications, each potentially built on a different version of .NET Core, without risking negative effects due to a different version of .NET than you built on. The downside to distributing an application like sdfg self-contained (i.e. shipped with all needed .NET Core files) is that the .NET Core files themselves are OS-specific. Hence, while we gain isolation and safety (no risk of .NET framework version issues), we also have to build per OS and architecture. I provide sdfg for 64-bit versions of Windows 10 and Windows Server 2016; if there is a significant need for another platform, contact me or feel free to clone the repo, add the needed support, and send me a pull request. To work in the sdfg code after you clone the repo, you will need Visual Studio 2017 with the latest update (currently 15.3) and the .NET Core 2.0 SDK - not just the runtime - installed. The DynamicFunc field discussed above enables you to write a C# anonymous function into the JSON runfile, and have it evaluated at runtime using Microsoft Roslyn technology. See SynDataFileGen.Lib/Factory.cs/GetFunky(). Currently, sdfg is limited internally to a maximum of 250 million (250,000,000) records per file. If you specify a value for FileSpec/RecordsPerFileMax that exceeds 250,000,000, it will be reset to 250,000,000.

Conclusion and Future

I hope the above discussion of scenarios where synthetic data is valuable, and of a synthetic data file generation tool I wrote for such scenarios, is helpful. Please contact me with questions or comments. If sdfg is useful to you, I would greatly appreciate it if you let me know. Similarly, if you clone it and improve the code, please send me a pull request! What's next for sdfg? A few ideas I am thinking about:

Enable runfiles to generate related files with cross-file references to generated values for categorical allowable value lists
Remove the per-file max records limit of 250,000,000
Implement a ContinuousNonNumeric field type to generate random non-numeric values without requiring a DynamicFunc code fragment
Add support for outputting files to other locations than the machine's file system. For example: Azure Storage

Separately: is there a need for a tool like sdfg that "understands" a relational database's schema and generates synthetic data in it, including respecting key relationships? Let me know your thoughts! Resources: The sdfg github repo: https://github.com/pelazem/syndatafilegen

The busy data scientist's walk-through to managing Azure Data Lake Store in R

Patrick El-Azem — Thu, 17 Aug 2017 13:10:09 GMT

Earlier this year (2017), my Microsoft colleague Ramkumar Chandrasekaran published a blog post about performing Azure Data Lake Store (ADLS) file system operations using the R language. After working through that for a recent project, I decided to write a detailed, step-by-step walk-through of getting this running, including the required Azure configuration steps.

Who is this for?

The audience is data scientists comfortable with R and R tools, who need to manage folders and data files in their organization's Azure Data Lake Store but without being Azure experts, requiring tools or languages they're unfamiliar with, or needing administrative access to their organization's Azure environment. To that end, while several of the steps below can be completed in multiple ways, I have chosen tools or technologies for simplicity (e.g. showing the Azure portal instead of automation scripts). The Resource Links listed at the end of this post include details on alternate ways to complete various steps, as well as code and other resources. In particular, the walk-through uses Server Principal Name (SPN) authentication, not a user account. This enables authentication and privileges to be customized for least privilege, and to be robust against organizational changes such as a particular user leaving the organization and their credential being disabled.

Why do this?

As organizations move to the cloud and undergo change, it's important to maintain productivity. Azure Data Lake is a fantastic platform for big data storage and analysis, but busy data professionals may not yet have the requisite tools or skills to work with new cloud technologies natively. Microsoft's cloud platform enables flexible and hybrid approaches to benefit from new technology while continuing to use existing tools and skillsets, allowing for transition and training at the right pace for people and organizations. This post was motivated by scenarios where organizations are storing production data in Azure Data Lake, but delivery timelines and operational/resource levels required simple, immediate access to the folders and files there for data analysts and scientists. There are certainly more robust usage scenarios for working with big data in Azure Data Lake; this post is intended to show how to quickly work with the Azure Data Lake file system without requiring clusters or extensive data movement orchestrations. To be clear: this is not a replacement for the analytical power of Azure Data Lake Analytics (ADLA) and U-SQL, or the Hadoop/Spark stack or other big data technologies.

What will we accomplish here?

By the end of this post, we will have completed Azure configuration steps and have a set of operational R scripts that can perform folder and file operations (upload, download, delete, list, rename) in an ADLS account. A data scientist will be able to work in their preferred R environment (such as R Studio or Visual Studio with R Tools for Visual Studio) and execute various R scripts against ADLS.

Pre-Requisites

To complete this walk-through, you will need an Azure subscription. Please ensure you, or someone working with you, can log into Azure before continuing. If you are not in the Owner role for this subscription, you will need to work with someone in that role to complete some Azure steps below.

To check what role you are in on your Azure subscription, in the Azure portal, click on the left nav menu, then "Subscriptions". Then click the subscription you will use.
When displaying your subscription, click "Access control (IAM)" on the left menu, then consult the blade to see what role you are in. As you can see here, I am grouping by Role and my account is shown in the "OWNER" group as well as showing "Owner" in the "ROLE" column.

Key Information

As you proceed through the following steps, you will need to collect and keep four items of information. Please note them as you encounter them:

The ADLS account name you create (I use "radls" below)
The Azure Active Directory ID
The Application ID for the Azure AD application registration you will create
A key for the Azure AD application registration you will create

As we encounter these items of information below, I will remind you to note them.

Overview

What will we accomplish in this walk-through?

We'll create an Azure Data Lake Store (ADLS) account in the Azure portal. *
We'll create an application and a Server Principal Name (SPN) in Azure Active Directory (Azure AD), also in the Azure portal. *
We'll grant the application appropriate permissions to our ADLS account. *
We'll apply initial file system permissions in the ADLS account. *
We'll install R pre-requisites.
We'll write R code to obtain an OAuth2 token, which we need to authenticate (as the SPN) when making calls to the WebHDFS REST API exposed by ADLS.
We'll write R code to run various file system operations against ADLS.

* These one-time steps require permissions on the Azure subscription. See below for details.

1 - Create an Azure Data Lake Store account

To complete this step, you will need to be in the Owner role for the Azure subscription you are working in.

1. After logging into the Azure portal, click "+ New", then "Data + Analytics", then "Data Lake Store / Create".
2. Provide the info needed to provision the ADLS account. (Note the ADLS account name you enter for later use, per "Items to Keep" above.) I set Encryption Settings to "Not enabled" for simplicity. Please set this to an appropriate value for your environment and data type. See Resource Link #2 for details. For testing, I recommend you create a new Resource Group and use it only for Azure resources related to this walk-through. When you are done, it's easy to de-provision the resources created for this walk-through by deleting just the Resource Group created here, which also deletes all resources deployed within it.)
3. Click "Create" and wait for the ADLS account to be created. Then, click into the resource group you specified in the preceding step to see the newly-created ADLS account. A new ADLS account's Firewall is not enabled by default. If you require Firewall protection, enable it from the ADLS account's "Firewall" blade, and set appropriate firewall rules so that users can later access the ADLS file system.

2 - Create a Server Principal

To complete this step, you will need to be in the Owner role for the Azure subscription you are working in. If you are not in the Owner role, you may still be able to complete this step: follow the steps at https://docs.microsoft.com/azure/azure-resource-manager/resource-group-create-service-principal-portal#required-permissions to determine if you have the needed privileges.

1. In the Azure portal, open the main left nav, click "Azure Active Directory", then click "New application registration".
2. Enter the info for the application registration, then click "Create". For "Application type", choose "Web app / API". For "Sign-on URL", you can enter a non-existent value. This will not be used in our scenario.
3. After the application registration is created, find it in the list of applications and click on it.
4. On the application's information blade, copy and store the value shown under "Application ID" for later use (per "Key Information" above). Then, click on "All settings" to open the application's Settings blade (this may already be displayed).
5. On the Settings blade, click "Keys". Create a new key by entering a key description and expiration. After entering these values, click "Save" to display the generated key.
6. After the key is created, the key value can be copied only once. Copy and note it for later use (per "Key Information" above) - do this immediately, as you will not be able to re-display this key.
7. Next, we also need the Azure Active Directory ID for later use (per "Key Information" above). Close the App registrations blade. Return to the Azure portal left nav, click "Azure Active Directory" again, then click "Properties" and note the value displayed for "Directory ID".

3 - Grant ADLS Permissions

To complete this step, you will need to be in the Owner role for the Azure subscription you are working in, or at least for the resource group you created above (the subscription owner can add you to the Owner role for the resource group).

1. In the Azure portal, navigate to the resource group you created. Click on the ADLS account you created, then click on "Access control (IAM)", then click "+ Add". We will grant the SPN permissions to just the ADLS account, rather than the containing resource group, to minimize its permission scope. Instead, you could grant the SPN permissions to the entire resource group; these permissions would then be inherited by all resources in the resource group. This would be useful for other scenarios with the SPN you created. However, I'll restrict the permission scope here and work only with the ADLS account itself.
2. Enter information for the permission we are creating. For "Role", specify "Reader". Then, select the name of the Azure AD application you created above; I used "r-adls-app" and have selected that here. Last, click "Save". The "Reader" role enables the SPN to see the ADLS resource, but not make any changes to it. Less restrictive roles (such as "Contributor" or "Owner") are not needed for this walk-through. The permission we are granting here applies to the Azure resource itself, not to the data within it (i.e. the files and folders we will work with; we will grant permissions for those in a later step). By assigning "Reader", we are preventing the SPN from being able to make changes to the Azure resource itself.
3. Verify that the permission was created correctly. After the previous step completes, verify that the permission was created correctly in the ADLS account's Access control (IAM) blade. Note that I am filtering to Reader roles only, for a more compact display. You may need to click "Refresh" and/or filter to quickly find the appropriate permission.

Intermission - verify you now have all Key Information

Before continuing, please consult the "Key Information" list above, and verify that you have copied and set aside all the required pieces of information for later use. Please treat this as sensitive information (especially the application key).

4 - ADLS File System Permissions

To complete this step, you will need to be in the Owner role for the Azure subscription you are working in, or for the resource group you created above, or at least for the ADLS account itself (the subscription owner can add you to the Owner role for the resource group or the ADLS account).

1. Navigate to the ADLS account and select its "Data Explorer" blade. As you will see, in a new ADLS account, no folders or files exist yet. I created a "/Data/Analysis/" folder structure for this walk-through. Consider a folder structure that makes sense for your scenario, and create it by clicking "New Folder". Navigate through the folder hierarchy you create by using the middle blade. Next, we will create file system access permissions to permit our SPN to perform all operations *only within the "Analysis" folder - not* in its containing "Data" folder**. This is to show a scenario where a data scientist may not have full access to the root of the ADLS account, and will be constrained to work within a subfolder.
2. In order for our SPN to access the "Analysis" folder, we have to grant the SPN "RX" (Read and Execute - not Write) permissions at each parent level in the containing folder hierarchy first, so that the SPN can traverse the hierarchy to reach our work folder. In my scenario, I have to apply appropriate permissions both at the ADLS root, as well as on the "Data" folder. Perform steps 2-4 here on both the root and on "/Data": select the folder, then click "Access". After verifying that the blade subtitle shows that I am at the correct folder (root or "Data (Folder)", shown), I click "+ Add" to open the "Assign Permissions" blade, where I will specify the user or group, and the permissions to grant. See Resource link 5 for details on ADLS file system access control permissions, and why RX permissions on the containing hierarchy are necessary.
3. In the "Assign Permissions" blade, click on "Select User or Group", then enter the name of the Azure AD application, i.e. the SPN. In this walk-through, this is "r-adls-app", as we have seen above. Once it is selected, click "Select" to return to the "Assign Permissions" blade.
4. In the "Assign Permissions" blade, click on "Select Permissions". Check "Read" and "Execute" Permissions. Set "Add to:" to "This folder". Set "Add as:" to "An access permission entry". Then click "Ok" to save this setting, then "Ok" in the "Assign Permissions" blade to finish saving the new permission. Remember - this is the enclosing hierarchy. These permissions are the minimal ones required to grant our data scientist full permissions in the contained subfolder. This is why we are not granting Write here, or setting this as a default permission entry. Remember to perform steps 2-4 both for the root as well as the "/Data" folder!
5. Similar to the previous steps, in the "Data Explorer" blade, navigate to the enclosed folder where the data scientist will work. In my scenario, that is the "Analysis" folder (full path: "/Data/Analysis/"). As done in the previous steps, click "Access" and "+ Add" to add a file system permission entry for this folder. Select the same user (the SPN) and continue to "Select Permissions". Check all Permissions. Set "Add to:" to "This folder and all children". Set "Add as:" to "An access permission entry and a default permission entry". Then click "Ok" to save this setting. Then click "Ok" in the "Assign Permissions" blade to finish saving the new permission. This time, we are setting permissions to allow the data scientist - who will authenticate using the SPN - full control in this folder and all folders and files contained within it - but not above it.
6. Back in the "Data Explorer" blade, ensure you have again selected the enclosed folder where the data scientist will work, then click "Refresh", then "Access". Verify that the permissions set in the previous step were correctly set and saved. If the permissions were set and saved correctly, this completes our work in the Azure portal for this walk-through.

Working in R

For the remainder of this walk-through, I will work in R Studio, with Microsoft R Client. However, please feel free to try this with your preferred R tool or IDE. Our R code will perform file system operations in our ADLS account by interacting with the WebHDFS API exposed by ADLS. We'll use the HTTR package to make the REST API calls. (Note that I'm using R since this was a specific requirement on the project I mentioned at the top; but any language that can interact with a REST API could be used for this part.) See the Resource Links below for more information on these topics. I have created a set of R scripts for the work described below. These scripts are available in my public github repo (see Resource Links). Please refer to this repo as I mention R files and scripts below. I recommend you clone the repo, or download the files, so you can work in the code while continuing below.

5 - R Pre-Requisites

We need one R package to complete this walk-through: HTTR (see Resource Links) and its dependencies. This is very simple to install:

install.packages("httr", dependencies = TRUE)

I have packaged this in the file 1-PreReqs.R in my github repo. Run this once in your R environment.

6 - REST API Authentication

In order to authenticate to the WebHDFS API using the SPN we created above, we need to obtain an OAuth2 token to send with each API request. Please edit security_info.R.

You will need to fill in the information (from "Key Information", above) as follows:

For security_adls_account_name replace with the ADLS account name you created above.
For security_tenant_id replace with the Azure Active Directory ID you copied above.
For security_client_id replace with the Azure AD Application ID you created above.
For security_client_secret replace with the Azure AD Application key you generated above.

After completing the edits, save and close this file.

The security_info.R file we just edited is used as a source file by security.R, which does the actual work of getting an OAuth2 token. This script exposes a security_get_token() function, which is called by the other R scripts we discuss below. This encapsulates all the token work for easy re-use across the various file system scripts. No edits are needed to security.R. Feel free to review it to see how it is getting the token and returning it from the security_get_token() function before continuing.

The R scripts provided in my repo for this walk-through follow a consistent structure. Each has these sections, delineated by comments: ### User specified # Security info # Execute In these files, you only need to perform edits in the section marked by ### User specified, for example to substitute in your own folder and file names. In each file, in the # Security info section, there is a call to the security_get_token() function in security.R, used to assemble the authentication header sent to the WebHDFS API in the # Execute section. The # Security info and # Execute sections are functional as is, and do not need editing for this walk-through.

Now that we have prepared our security_info.R file with the info we created earlier in this walk-through, and reviewed our security.R file, we are ready to start working with the ADLS file system from our R environment!

7 - ADLS file system operations

We'll start by listing folders in our ADLS file system. Recall that, above, I created a "/Data/Analysis/" hierarchy. If all the previous steps worked correctly, I should now be able to list folders in my ADLS file system. To do this, I'll run 2-ListFolders.R. Note that on line 6 (see last screenshot in the previous section), I'm setting the folder name whose contents I want to list. Since I'm listing the contents of "/Data", I should see the "Analysis" folder in the results.

The output shows our "Analysis" directory, so we have successfully queried the ADLS file system!

Now let's upload a file into our "Analysis" folder. In 5-UploadFile.R, I've specified a file on my local system (d:/test/test1.txt), and a destination path for it in ADLS (/Data/Analysis/test1.txt). Edit these values as needed. If the upload succeeds, this script should output an HTTP 201 status code at the end, and the file should show up in the ADLS account's Data Explorer display in the Azure portal.

The R Studio console output shows an HTTP 201 status code, meaning success - the ADLS Data Explorer in the Azure portal shows the file uploaded successfully! We have now used both our Read and Write permissions to read from, and write to, our ADLS file system.

Please explore the remaining R scripts in the github repo and test them. In particular, 8-ReadFile.R allows files to be downloaded to the local system, so that data scientists can work on source data files in their local environment and with their local tools, as needed.

Conclusion

In this walk-through, we started with only an Azure subscription. We created an ADLS account and an Azure AD application registration, and configured security for R scripts to run against the ADLS file system by interacting with the WebHDFS REST API. I hope this walk-through is useful to you. Please contact me with questions or comments - and pull requests are always welcome!

Resource Links

For more information on pertinent topics in the walk-through, please refer to these sources:

Ramkumar's post: https://blogs.msdn.microsoft.com/microsoftrservertigerteam/2017/03/14/using-r-to-perform-filesystem-operations-on-azure-data-lake-store/
Getting started with Azure Data Lake Store: https://docs.microsoft.com/azure/data-lake-store/data-lake-store-get-started-portal
Create an Azure AD app and Service Principal: https://docs.microsoft.com/azure/azure-resource-manager/resource-group-create-service-principal-portal
Service-to-service authentication, ADLS with Azure AD: https://docs.microsoft.com/azure/data-lake-store/data-lake-store-authenticate-using-active-directory
ADLS access controls: https://docs.microsoft.com/azure/data-lake-store/data-lake-store-access-control#common-scenarios-related-to-permissions
ADLS REST API: https://docs.microsoft.com/azure/data-lake-store/data-lake-store-get-started-rest-api
Hadoop WebHDFS REST API reference: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html
CRAN HTTR reference: https://cran.r-project.org/web/packages/httr/
My github repo containing the R scripts used in this post: https://github.com/pelazem/azure/tree/master/adls-r
The Azure portal: https://portal.azure.com

Azure CDN: overview and dynamic content

Patrick El-Azem — Mon, 13 Mar 2017 10:55:39 GMT

What is the Azure Content Delivery Network (CDN)?

The Azure CDN offers a global solution for delivering content to web browsers with minimal latency by caching publicly available assets such as image, CSS, or script files at geo-dispersed edge nodes. Web pages refer to assets (such as image files) by their CDN URLs. Referring to a relative URL (for example, ) means the web browser has to download the asset from the source web site. This may not be optimal if the web site is hosted far away from the browser, or if the web server is heavily loaded. Instead, using a CDN lets asset files be downloaded from edge nodes (dispersed web servers that store copies of the asset files), which both offloads the source web site and shortens the time for web browsers to download files, as there will likely be an edge node closer to the web browser's location than the source web server. This means page load times are reduced, resulting in a more responsive user experience.

Getting content to the CDN

How does the CDN get the assets that it distributes to its edge nodes? When setting up a CDN, an endpoint is defined, and an origin must be specified for the endpoint. The endpoint defines the hostname that will be used to refer to assets, and the origin defines the asset source. For example: let's say I define an endpoint called mycdnendpoint. The hostname for asset retrieval would then be mycdnendpoint.azureedge.net (the azureedge.net top-level domain is fixed for Azure CDN). Now, I need to define an origin from which the CDN will get assets for the endpoint. In Azure CDN, I can choose from these origin types: Azure blob storage, an Azure cloud service, an Azure web app (this lets me use Azure services exposed as web apps, such as API apps or Azure functions), or any HTTP/HTTPS URL for maximum flexibility. Choosing Azure blob storage as an origin means it's very simple to expose assets over the CDN: simply load all the assets into the chosen blob storage account (folder structures are fully supported), append the path to the endpoint hostname, and construct a URL. For example: I have an image called microsoft.png in an assets container (root folder) of the Azure storage account at which I pointed a CDN origin. My CDN URL for this image would be https://mycdnendpoint.azureedge.net/assets/microsoft.png. When any web browser loads a page that includes this image URL, the browser will contact the Azure CDN to download that file, and the CDN will redirect my browser to the CDN edge node that is closest to me to download the file. Here's a conceptual diagram showing CDN elements, including content sources, CDN origins and endpoints, and public-facing web sites that include CDN-served content. [caption id="attachment_165" align="alignleft" width="814"] Conceptual diagram showing CDN origins, endpoints, and consumption.[/caption] I'm leaving out a few details here, such as CDN support for custom domains (i.e. use your own top-level domain name instead of azureedge.net), choosing either or both HTTP and HTTPS, using only a subset of a storage account for CDN assets, using host headers, caching on query string combinations, and more. These beyond-basic capabilities will likely be part of any production-class CDN rollout. Please refer to the Azure CDN documentation for more details.

Cache Misses and Hits

Let's say I just updated my web site with some new pages. These refer to new image files that I just uploaded to my origin blob storage. How do these images get out to all the edge nodes so that visitors to my site see them? CDNs typically do not pre-load asset files to all the edge nodes. This would require a prohibitive amount of network traffic and edge node storage (after all, each edge node would have to cache every single file in every origin). Instead, when an asset file is requested from an edge node for the first time, the edge node will check its own file cache. If the asset file isn't there - known as a "cache miss" - the edge node will request the file from the origin, cache it locally, and then return it to the web browser. This means that the first person to request a file will incur a delay similar to requesting it from the source; however, the edge node now has the file, and will keep it cached locally for later visitors, who will get the asset directly from the edge node without going back to the origin, known as a "cache hit".

Time To Live (TTL)

How long does the edge node keep the file cached? Typically, a file has a time-to-live (TTL). If a file is not requested by anyone during that TTL, the edge node deletes the file from its cache (but not from the origin, of course). This helps manage edge node resources. TTL is customizable by several methods; if none is set on an asset, Azure CDN defaults to a TTL of seven days. The appropriate TTL will depend on factors specific to your scenario. For example, your organization's logo presumably does not change very often, but gets downloaded very often as it's on every page of your web site; therefore, consider a longer TTL so that edge nodes don't retrieve this file from the origin more often than is needed. By contrast, files that change periodically should get shorter TTLs so that your users don't see outdated files (arguably, deploying updated asset files with new file names is a surer way of ensuring your site visitors see the freshest content, but this may not always be practical). There are further considerations to setting the "right" TTL, including balancing network traffic between edge nodes and the origin vs. content freshness; the Azure CDN includes a number of reports (such as cache hit ratio) that can help you tune your TTLs. [caption id="attachment_76" align="alignnone" width="725"] Azure CDN Cache Hit Ratio metric[/caption]

Timeline

What does this look like in reality? The following screenshot shows repeated loads of an image from an Azure CDN endpoint in my web browser's developer tools network timeline. The first download, labeled as "First load (CDN cache miss)", shows a total time of 650ms. This is because the edge node has to retrieve the file from the origin, which takes longer. Later loads, labeled as "Later loads (CDN cache hits)", have much shorter total times, between 16 and 49ms. This indicates that the CDN edge node now has the file in its cache and doesn't need to go back to the origin to get it. (Note on the screenshot that I have disabled my browser's local cache, so that the CDN's cache effect is obvious.) [caption id="attachment_75" align="alignnone" width="776"] Asset first and later loads from CDN edge[/caption]

New content to CDN immediately

So far, we have discussed straightforward CDN provisioning from an origin with a relatively stable set of asset files, presumably deployed when a site updates or on another relatively predictable interval - possibly even a continuous deployment process in an ALM solution like Visual Studio Team Services. But what about scenarios when content is generated dynamically, for example in a browser-based image editing tool, and that content needs to be immediately available through a CDN? Clearly, waiting for a manual upload or publish process, or a periodic continuous deployment process, would not be suitable due to the delay between asset generation and CDN origin availability. In such a scenario, we need to get the newly created asset file into the CDN origin right away, ideally as soon as it is created.

File-based content

Azure blob storage is ideal as an origin for static files. When a file is generated by some sort of process and needs to be CDN-available right away, we can write the file to our Azure storage programmatically as soon as it's created. There are Azure SDKs in various languages and stacks. Uploading a file to Azure blob storage and setting its TTL is straightforward using the Azure storage SDK. The following screenshot shows the pertinent code; I'm using C# but again, there are SDKs in various languages. Please see the Resources section at the end of this post for a link to the full code base. On lines 142-143 I'm creating a blob reference for the file I'll upload and setting its content type (this is a MIME type designation; for an image file like a .png or .jpg, the content type would be image/png). Setting the content type here ensures that it propagates to the edge nodes and lets them serve the content up correctly to web browsers. On line 146, I'm setting a retry policy. This is optional, but good practice; it instructs the Azure blob storage client I'm using to upload my file what to do in case the upload is interrupted. Here, I'm telling it to wait five seconds, then retry up to a max of five times. Guarding against transient failures is a highly recommended best practice in cloud application development. On line 149, I am asynchronously uploading the file to my blob storage account. On line 152, I set the blob's CacheControl property; this is what CDN edge nodes will use for the file's TTL in their cache. The TTL value is an integer and is measured in seconds. On line 155, I'm updating the file's properties with the TTL. Then, on line 157 I get its URI in Azure blob storage - this is not a CDN URI yet, but I'm retrieving it so I can first check that it uploaded successfully to blob storage. [caption id="attachment_95" align="alignleft" width="902"] C#: upload file to blob storage , get URI.[/caption] The blob storage location to which I uploaded this file is the origin for my CDN endpoint, so at this point the file should be available not only through its direct storage URI (which I retrieved on line 157, above) but also through CDN URLs. To confirm this, I use an IsBrowsable() method to check that the file I uploaded is available directly from its Azure storage URI as well as through my CDN. Note that I confirm both HTTP and HTTPS CDN URLs - having both available is useful to avoid browser warning messages and lets web developers avoid hard-coding the URL scheme. [caption id="attachment_115" align="alignleft" width="888"] C#: check blob storage and CDN URIs for uploaded file.[/caption] [caption id="attachment_125" align="alignleft" width="923"] C#: IsBrowsable() method to check whether a URL is available[/caption] The preceding code fragment (see Resources at the bottom of this post for a link to the complete code) shows a file being uploaded to a blob storage account that is used as a CDN origin, with nothing else that needs to be done to make the file available through the CDN. So this is a very straightforward way to generate files and immediately expose them through the Azure CDN.

Stream-based content

Sometimes we may need to serve content that isn't easily persistable as a file in a storage repository; for example, we may need to return a content stream. In the following code fragment, I have a REST API method that instantiates a content stream and writes it to the HTTP response's content property. The method also sets the TTL on the HTTP response using the CacheControl header property; here, I'm setting the TTL to 10 minutes (600 seconds). [caption id="attachment_135" align="alignleft" width="996"] C#: API method HTTP response with stream content[/caption] A REST API that returns content streams lets us use an Azure CDN endpoint that points at the API, rather than at a static storage location. Using an API lets us accommodate a very wide variety of dynamic content-generation scenarios, including varying by parameters or query string, geography, time of day, and more. Using an API URL like https://mycdnapiendpoint.azureedge.net/api/content/i3 (see Resources at the bottom of this post for a link to the full code for this API app), we can again see the pattern of first load being slower than subsequent load as the CDN edge node first has a cache miss, retrieves the content from the origin, and then on subsequent requests has a cache hit. [caption id="attachment_145" align="alignleft" width="664"] CDN API URL first and later loads[/caption]

Conclusion

The Azure CDN is an excellent way to offload your web application of serving static content. It offers lots of advanced features, a global footprint, and supports multiple endpoint types to accommodate static and dynamic content scenarios.

Resources

These are, respectively, links to apps that upload files to Azure blob storage and check immediate CDN availability (Storage Origin App), and a REST API app that returns a content stream based on an ID parameter. Please note the disclaimer here, and feel free to reach out to me with any questions.

Get started with Dynamic Data Masking in SQL Server 2016 and Azure SQL DB

Patrick El-Azem — Mon, 02 Jan 2017 16:40:11 GMT

Dynamic Data Masking (DDM) is a new security feature in Microsoft SQL Server 2016 and Azure SQL DB. The main documentation is here (also see link under Resources at end). This post is a quick how-to intro to DDM, including applying it in a database and managing which principals see masked or unmasked data. I'll also answer a few questions that commonly come up.

What is DDM?

Picture this scenario. You have a database table which stores sensitive data, such as social security numbers, in the clear (unencrypted). Anyone with appropriate access can run select * against this table and see all the sensitive data. This becomes a concern in organizations where production data is periodically restored into development, test, and/or staging environments. Developers, testers, and other people need to work with the data, but have visibility to sensitive data. This is clearly concerning (and may be unlawful in some jurisdictions). How do we give these roles the data they need, while protecting sensitive data? This is what DDM does. A database administrator applies a masking rule to a column. The result is that when developers, testers, and other non-administrators run queries against the table, they no longer see masked columns' data in the clear; instead, they see the data in obscured format. This does not mean the underlying data is now stored encrypted. No - it's still stored in the clear, and any role with UNMASK privilege can still access the data in the clear. DDM is simply a way to obscure sensitive data from roles or users who should not see it in the clear.

DDM in action

Let's say you have a table with social security numbers. A select * returns this: We want to use DDM to obscure this data. How do we do that? Simple. We use familiar alter column syntax to designate a column as masked, and we specify how to mask it. Here's how we'd mask the SSN column in T-SQL:

alter table data.Users
alter column [SSN] add masked with (function = 'partial(0,"XXX-XX-",4)');

What's happening here? First, we're designating the [SSN] column as masked (add masked), then we're specifying how to mask it, using a masking function. Currently, four masking functions are available; I'm using partial(), which lets me specify how many characters to leave clear at the start, what string to substitute in the middle, and how many characters to leave clear at the end. After applying this mask, the same select * returns this: Note how the SSN data is now masked. Per our masking function, no characters are clear at the start; the data is partly replaced by the mask we specified; and the last four characters are in the clear. How do we remove masking from a column? Simple:

alter table data.Users
alter column [SSN] drop masked;

Note that DDM works in T-SQL for both SQL Server 2016 and Azure SQL DB. However, for Azure SQL DB, DDM can also be configured using the Azure portal.

DDM Security

By default, all principals not in the db_owner role will see masked data; principals in db_owner are automatically granted UNMASK privilege and will see data unmasked (in the clear) regardless of any masking rules applied. DDM does not prevent masked users from executing DML statements. For example, if they have DELETE privilege, they can delete records from a table with a masked column; similarly with UPDATE. This is expected.

DDM FAQ

Why would we use DDM if it doesn't even encrypt the underlying data? Because in many scenarios, such as dev and test environments, masking is enough protection without the performance and administrative overhead of encryption. For example, daily restores from production to a staging database used for manual and/or automated tests where it's important to understand the underlying data. In any case, SQL Server offers excellent encryption options. DDM is a great option when masking is enough.

"We have some people who need to see masked data in the clear. Does that mean we have to add them to db_owner? We'd prefer not to." No, people do NOT need to be added to db_owner to see data in the clear. Specific roles or users can be granted UNMASK; currently, this is at a database level, i.e. not to a specific table or column. Here's how to do this for an OpsMgrRole database role: grant unmask to [OpsMgrRole];

What if someone selects masked data into a temp table, or backs up the database and restores it on another machine? Can they get around DDM on the source table this way? Short answer: No. Longer answer: When a user to whom masking applies selects into another table or backs up the database, DDM will write the masked data statically, i.e. in this instance, the written data is no longer "in the clear"; what is written is statically masked. Let's prove it with this T-SQL: What's happening here? On line 4, I'm reverting to my logged-in user (which happens to be a db_owner member). This is to make sure I am not in the context of a less-privileged user. On line 7, I'm impersonating a lesser-privileged user who is not a member of db_owner, and who has not been granted UNMASK. On line 10, I'm creating a temp table. On line 13, I'm extracting data from the source table, which has a mask on the SSN column, into my unmasked temp table. By doing this, I'm trying to get around the mask on the source table. On line 16, I'm preparing to see people's sensitive data... only to be disappointed - the data is still masked! In fact, the data was written to the temp table statically masked, i.e. it's no longer in the clear in the temp table - it was written with all the X mask characters. On line 19, I'm reverting back to my logged-in user, who is a db_owner member. On line 22, I'm selecting data from that temp table again, but (as expected) it's still masked. Remember - it was written to the temp table statically masked by a user subject to masking on the source table, so my db_owner membership and implicit UNMASK privilege do not help. Last, on line 25 I query the source table and see the data unmasked, exactly as expected, which demonstrates that the source table is dynamically masked whereas the temp table, as expected, is statically masked. The three result sets from lines 16, 22, and 25 in the above T-SQL: Note: the above T-SQL is a screenshot image. I used that for easier readability with color and line numbers. See Resources below to get all the code I used for this post.

Will DDM interfere with query execution plans? No. Queries are executed first, then the results are masked. Nothing changes with regard to query execution plans.

We have a more complex situation that's not covered by the currently available masking functions. For example, we have variable-length fields that store people's comments and notes. Can we detect, and mask, social security numbers or credit card numbers within such text blocks, but not mask anything else? So something more like a regular expression. That's not possible at this point with DDM, but the SQL product group has gotten this feedback.

Resources

DDM MSDN documentation: https://msdn.microsoft.com/library/mt130841.aspx My code used for this post is in my public git repo (please first note disclaimers at the repo root - this is sample code, use at your own risk). The readme for these files contains details, but briefly, I provide you code to create a sample database and then implement and test DDM, including the DDM subvert fragment above. All you need is a SQL Server 2016 test instance. Hope this helps you get started with Dynamic Data Masking.