Prerequisites:

1. Docker for Windows (this should work the same on Linux as well with some slight modifications of source paths & maybe syntax)

Optional:

1. Visual Studio Code
2. SQL Server Management Studio (may not be needed if you use mssql extension for vscode).

Install

1. Install Docker for Windows. This is a simple wizard install. Make sure you have enough memory to run SQL Server in Docker – see this: https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-docker

Optionally install the other software. Technically you could run the TSQL code from sqlcmd inside the Docker container but in my setup, I don't usually do that because I want to externalize any data & code from the running container that way I can easily upgrade to new releases of SQL Server as they come and do things like check in my code to a code repository. The rest of the steps can be done when you want to run an instance and serve data but making sure Docker is setup and operational is a good thing to have ready to go.

Start Docker and Run the container

I usually keep Docker turned off and start it up when I need to run a container.

1. Launch Docker and wait about 30 seconds for it to start up.
2. Launch vscode and open the PowerShell script with my saved commands.
3. Run the commands per the script (except the last one). Note that the first time you do the pull it may take a minute or two to download the container image. When you run this the second time docker will check to see if you have the proper image or if a new one needs to be downloaded, if not then docker will save you time and not download the image again.

   
   # Confirm that Docker is running
   docker ps

   # Start container

   docker pull microsoft/mssql-server-linux:2017-GA

   docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=***************" -p 1433:1433 --name sql1 -d microsoft/mssql-server-linux:2017-GA

    

   # Confirm that the container is running. SQL Server should be up and running now and ready to host data

   docker ps

    

   # Copy sample DB MDF file into container, make sure you change the attach database files to location:

   docker cp C:\Users\username\SampleData\AdventureWorks\AdventureWorksLT2012_Data.mdf sql1:/var/opt/mssql/data/AdventureWorksLT2012_Data.mdf

1. Open the .sql file containing the code needed to attach the database and run the command:

   -- Attach the MDF file copied into the container

   USE [master]

   GO

   CREATE DATABASE [AdventureWorksLT2012] ON

   (FILENAME = N'/var/opt/mssql/data/AdventureWorksLT2012_Data.mdf')

   FOR ATTACH;

   GO

    

2. Open console in vscode by pressing Ctrl + Shift + P
3. Type "connect" and you should see the option MS SQL: Connect, select it and press Enter.
   1. Follow steps to create a connection profile for this locally running SQL Server, once saved you'll just select the profile
4. Select the code in the script and execute it by pressing Ctrl + Shift + E
5. You might see response saying that the database has been upgraded in version (I'm seeing upgrade from version 864 to 863 for my MDF file) but as long as the MDF file is in working order the attach should work as planned.
6. To confirm the attach run SELECT name, database_id, create_date FROM sys.databases; and you should see your database listed there. From that point you can run USE command and query as needed.
7. When finished close all client connections to the server and then go back to cleanup your docker container by running the last line in the script:

   # When finished with SQL Server, use this to stop the container

   docker stop sql1

So, there you have it, SQL Server on Linux on Docker on Windows in 10 easy steps. These very simple scripts are saved in this GitHub repo but it's probably easier to just copy and paste or memorize.

Part 1: An Introduction
Part 2: Capabilities and Components
Part 3: Storage & Processing at Scale (you are here)

Hadoop provides many benefits when it comes to storing substantial amounts of data as well as providing a framework for efficient mass-scale processing of data stored in the storage layer of a Hadoop cluster. Given the popularity of Hadoop and the foundation of innovation in the Hadoop ecosystem, we have many tools we can use to meet many needs, using tools besides Hadoop but often using tools beyond Hadoop.

Much of the information below is not new and I'll only go into it to ensure all readers understand the value components like HDFS & YARN can bring (remember capabilities from the last post in this series?), which is critical in the conversation about running them in the cloud. Why run ANY technology system ANYWHERE if there is no clear value coming from it? I'm going to assume some knowledge of Hadoop is known but if anything is not clear, please ask questions in the comments and I will refine.

HDFS

The Hadoop Distributed File System (HDFS) provides a storage environment that allows for any data type of virtually any size, type or characteristic in a general-purpose file service. There are some limitations of the features, for example small files aren't typically stored efficiently in an HDFS as well as the read-only nature of the file system that can complicate some scenarios but many times these are solved for in the larger architecture of a solution or implementation (out of scope in this case, you'll understand why in the next post).

I would say that HDFS is mostly what enables the traditional ETL workload to become an ELT workload. ETL stands for Extract, Transform and Load which was a widespread practice in data warehousing and analytics in previous decades. It involved reading data from sources, transforming it (either in an expensive data store like a data warehouse appliance or at least an SMP RDBMS that used expensive storage) and then writing it out to a data store that was consumable for analytics purposes by a business unit that needed answers yesterday. ELT enabled a change in workflow that allows the same process to unlock value for the business, answer questions but do so in a more flexible and agile fashion. For example, in the ETL method one always has to know the structure of your data before you can write it into or transform it in the expensive RDBMS while ELT at least allows you to get datasets into a store and enables exploration of data beyond samples.

HDFS like many enterprise technologies has been built with the expectation that hardware will fail. The solution in HDFS was to make multiple copies of the data (usually 3 by default) and this usually allows inexpensive disks to be used to store data rather than typical expensive disks which occupy Enterprise-class storage systems. Multiple copies of the data are also key in giving the option for any processing that needs the data to have the option of fault tolerance, you can't continue processing data if there is only 1 copy of the data and a cluster node storing that data goes offline.

Another key capability of HDFS is that data is usually split into multiple chunks of data to allow reading of that data (and subsequent transformation or processing) on multiple nodes. Since we're already storing data on multiple computers why not also process data on multiple computers, so we can breakup large processing jobs into smaller jobs running on as many nodes in the cluster as possible. When taking this approach overall processing time goes down. That brings us to negotiating resources across many computer nodes in a cluster.

YARN

When operating large-scale processing across many computers it can be a challenge to ensure the processing running on that cluster is operating efficiently. Yet Another Resource Negotiator (or YARN for short) was created to give Hadoop operators the capability of maximizing the resources across a cluster of computer nodes that can both store and process data. YARN allows different workloads to run across the same cluster of storage and compute resources and each have fine-grained control over the resources available for processing. For example, if a Hadoop cluster is used by the data warehousing team for reading data from source line of business data stores and writing into an enterprise data warehouse the administrators of the cluster can give them 70% of the overall compute resources (CPU cores & memory) while also ensuring the application R&D department has 30% of the compute resources for their needs as well. When viewing at this level of control it is almost a cloud-like capability when it comes to elasticity (although when deployed on premise a Hadoop cluster does have a top-end compute and storage capacity, more on cloud soon).

Both YARN and HDFS provide capabilities that otherwise are hard to come by and are relatively straightforward and easy to understand. The architecture of the services are simple and often involve Master & Worker roles that ensure the cluster is operational and can provide necessary capabilities like fault tolerance, scalability, resiliency and other Enterprise requirements.

Next time I'll will discuss cloud concepts like virtualization, IaaS and PaaS and why organizations find them valuable.

Part 1: An Introduction
Part 2: Capabilities and Components (you are here)
Part 3: Storage & Processing at Scale

Capabilities

Whenever I talk to a customer or look to solve a challenge I always think it's best to start with a discussion around capabilities. Capabilities speak to the actual needs and help us not get caught up in a technology discussion or trying to solve the challenge before we fully understand the challenge. When we need to transform large amounts of data or process through large amounts of data in preparation for business centered objectives knowing the capability needs is essential. This is not intended to be a complete or exhaustive capability list.

Transforming, querying or processing large amounts of data means storing large amounts of data (also need to know other characteristics like where it's coming from, the nature of data movement, security requirements, etc. but we'll start with storing). Historically this was done in either a file system or some kind of database or both. Storing data is only valuable if you can also retrieve the data needed when you needed it so relational databases were created to make this easier. As data volumes grew things like columnar store databases and massively parallel processing databases were created that eased some growing pains of changing data characteristics. All along capabilities grew and changed and new data stores arose included distributed databases and distributed file systems. From a capability perspective, we still need to store (or write) data to a store and we need to retrieve (or query or read) data from a store. As complexity grows & technologies change these capabilities are still constant.

Another key capability is the retrieval of data from a data store, with Hadoop that means MapReduce or YARN… or Hive, or Pig, or Drill, or Spark. This is a growing list, but Spark is a general execution engine that has been successful from an industry adoption point of view, it's a de-facto standard. YARN is the backend service that provides resources for processing and in most cases services like Spark will create a YARN application to get the work done in a distributed fashion. I like to think of YARN as an interface for which applications can be executed on a cluster of resources.

Other capabilities most organizations need include high availability and fault tolerance. I won't deep dive into them, but they are equally important as massive scalability. This is not an exhaustive capability list.

Components

After understanding the capability needs to solve a challenge then we can start looking at ways we can meet those capabilities via services or tools (only look for a hammer if you need the capability of using a nail, it might not help much if you need to use glue). For example when there is a need for a capability of storing petabytes of data one might consider the most simplistic route to build a computer/server that can store petabytes of data, maybe even something like the Titan in use at Oak Ridge National Laboratory which can store 40 PB of data (which is good) but costs $97 million and takes up over 4,300 square feet of floor space. Most organizations don't have that kind of budget or want to give up that much floor space. Since more and more organizations are seeing a need for storing volumes of data closer to that size but don't have that kind of budget (remember there are also people costs in maintaining an infrastructure like that) or expertise in system of that scale other options need to be explored. There are other distributed file systems and distributed data stores that might make sense. Over the past decade Apache Hadoop HDFS has been the star of the distributed file system game, although others (Lustre, GlusterFS,etc.) are popular in some circles. HDFS made it possible to buy inexpensive server hardware and achieve capabilities like massive scale (many petabytes), high availability & fault tolerance and other capabilities all through the application layer (re-read this sentence and mentally highlight inexpensive hardware).

In the next post, I'll go more in-depth on HDFS and YARN and how is meets the capability needs and how those can be implemented in the cloud. As always – please share any thoughts or feedback below.

Note: In an effort to share more often and react to feedback I might receive, I'm going to break this up into a series of posts rather than writing a book.

Part 1: An Introduction (you are here)
Part 2: Capabilities and Components
Part 3: Storage & Processing at Scale

The origin story

Years ago, I worked with a company that was implementing Hadoop as a means of speeding up data transformation processes on petabytes of information received from client organizations. We had used traditional ETL for years before massive datasets & increasing business demands meant we needed to look at other options. Through many meetings, I introduced Hadoop to our technical data & infrastructure teams and we started down the path of setting up some initial POC clusters. Eventually we had environments & architectures designed that solved development & test, production as well as NoSQL-based solutions for indexing & searching decades of communication data as well as transactional data for major retail customers in US, Canada, Latin America, Europe, Asia and Australia.

Hadoop & why parallelism is great

At the core of our solution was a traditional Hadoop cluster utilizing the Hadoop Distributed File System (HDFS) for data storage paired with MapReduce-based processing in our case this was Apache Hive queries. This worked great as we took some ETL workloads that took 160+ hours down to less than a day. It also gave us improved data management capabilities as appending hundreds of GB of data to existing tables now was as easy as uploading new files to existing HDFS directories thanks to the schema-on-read nature of Hadoop & Hive.

Since our on-premise and co-lo data centers were the only data centers we used we went from POC to Production with physical servers that lived in a rack in our datacenter and we built backup & disaster recovery strategies around dual-pipeline processes & cross-cluster data replication from Dev/Test & Prod clusters to a DR cluster in a DR location in another state. We had a short-lived experiment that used virtual machine nodes but were quickly convinced by a Hadoop vendor that physical was best and in those days, it was hard to argue that VMs would provide a better solution.

HDFS & Hive-on-MapReduce allowed us to take large datasets and introduce parallelism in our processing very easily. We didn't have to think about partition keys or segmenting data within our cluster, we copied a 50 GB data file to the cluster and HDFS split the file into blocks for us and we could trust that when we ran our Hive query against that data there would likely be a degree of parallelism (though sometimes this was very small until we did think about partition keys and data segmentation) and we'd likely see a performance gain. At the end of the day we implemented Hadoop, saw impressive performance gains and everyone was a hero to the business.

And then the cloud happened…

After I helped the company establish solutions to priority challenges and I felt that team was affixed on a solid path I moved on to other challenges. At the same time, the rest of the world had been, and was awakening to, this thing called the cloud. Architecting a Hadoop-based solution in the cloud should look different than one on premise and the purpose of this series is to dive into those reasons, explore what they mean and ultimately prove out why I think Hadoop will evolve into only an API (if it hasn't already).

A great feature of Power BI Pro is the Gateway. Power BI Gateway comes in 2 variations, Personal & On Premise. Personal is just that – used for testing on a developer computer and only runs when the user is logged in. On Premise is used run under a service account on a server and has some extended capabilities for live or Direct querying of SQL Server/Analysis Service data sources.

I work with healthcare customers and many times there is a need to plan a hybrid cloud strategy involving both on premise and Azure. I recently worked with a customer to use Power BI for internal reporting and dashboards with both Azure-based and on premise-based data sources.

Here are the steps used to stand up Power BI On Premise Data Gateway and create reports using an on premise MariaDB data source (MariaDB is an open source database service that is "drop-in compatible" with MySQL).

Install the MySQL Connector/Net, ADO.NET Driver (to be completed on the gateway server)

Depending on your setup, this might already be available on your system. You should just need the driver, not necessarily the MySQL for Visual Studio integration tool. I was directed to download the driver from the MySQL site here: https://dev.mysql.com/downloads/connector/net/6.6.html

Setup the Power BI On Premise Data Gateway (to be completed on the gateway server)

1. Login to https://www.PowerBI.com with your work/Pro credentials.
2. In the upper right-hand side of the page click the download menu and select Data Gateway

   

    

3. Download the software and run the install.
4. Click Next.
5. Select On-Premises Data Gateway (not Personal Gateway) during the install.

   

    

6. Click Next.
7. Click Next.
8. Confirm location for software to be installed & confirm terms of use. Click Install.
9. You should see confirmation that installation was successful. Click Sign In to continue. Sign in with your workplace/Power BI Pro credentials.
10. Select Register a new gateway on this computer. Other option is for cases when you might need to migrate an existing gateway to another server for example.

    

     

11. Give your new gateway a name and provide an 8-character recovery key:

    

     

12. Click Configure.
13. Confirm that the gateway configuration completed and is ready to use with a message like this:

    

14. Click Close.

Add a Data Source to your Gateway (to be completed on any computer with a browser)

1. Login to https://www.PowerBI.com with your work/Pro credentials.
2. In the upper right-hand side of the page click the download menu and select Manage Gateway.

   

    

3. Your new gateway should be listed by name on the left-hand side. Click ADD DATA SOURCE to add a new data source to the gateway.

   

    

4. Enter the details of your new data source:

   Data Source Name

   Data Source Type (select MySQL)

   Server

   Database

   Authentication Method (Basic or Windows. For my testing, I'm using a user account managed on the MariaDB server, so I need Basic)

   Username

   Password

    

   

    

   Advanced Settings are used to set the Privacy level of this data source. On-premises data gateway can be used by users across your organization so it may be important to set the Privacy level of data sources using this setting.

    

5. Click Add. The gateway service will confirm that it can access the data source you are attempting to add.
6. Click Users tab.
7. Add users to the data source by entering their address. Click Add. This is also where you can remove users.

Create your Power BI report/PBIX (to be completed with Power BI Desktop)

If you've never used Power BI to create a report using an on premise data source, you can use these articles to walk you through the process:

Get started with Power BI: https://Power BI.microsoft.com/en-us/documentation/Power BI-service-get-started/

Create a new Power BI report: https://Power BI.microsoft.com/en-us/documentation/Power BI-service-create-a-new-report/

Power BI – basic concepts for Power BI service: https://Power BI.microsoft.com/en-us/documentation/Power BI-service-basic-concepts/

You don't need to tell Power BI that you will be connecting to the on premise data source via a gateway, just connect directly to the source. When you publish the PBIX file, the Power BI Portal will handle sourcing the data through the gateway for published reports.

Publish Power BI report (to be completed with Power BI Desktop)

1. Starting with a Power BI report that is ready to be published.
2. Click the Publish button in the Home tab of Power BI Desktop

   

    

3. Select publishing destination, likely you'll want to publish to your Power BI Workspace ("My Workspace").
4. Power BI Desktop will publish the PBIX file to your Workspace in the Power BI Service. You should receive confirmation with a link to the published report.

Appendix

Ensure proper gateway is selected for the data source

Depending on your setup you may have other gateways setup in the Power BI Service. In order for data updates/refreshes to flow through from an on premise source like MariaDB to the Power BI Service you'll need to ensure the proper gateway is selected.

1. Login to the Power BI Portal: https://www.PowerBI.com with your work/Pro credentials.
2. In the left-hand navigation menu scroll down to the Datasets section.
3. Click the … button for your newly published dataset (should be the same name as your published Power BI report).
4. Click SCHEDULE REFRESH, this will open the report preferences.

   

    

5. Datasets tab should be selected, click Gateway connection and then ensure the proper gateway is selected. You will also notice a status for each gateway

   

    

6. Click Apply if changes are made.
7. Initiate a manual refresh by selecting REFRESH NOW from the dataset menu.

Schedule refreshes

Once a Gateway connection is setup (per instructions above) you should be able to initiate a manual refresh of data from the on-premise source. You'll also be able to keep data in your workspace up-to-date via daily or weekly refreshes.

1. Click Schedule Refresh under the Datasets tab.
2. Change the selector to Yes for keep your data up to date.
3. Select a refresh frequency (Daily or Weekly).
4. Select a time zone.
5. Select a preferred time for the refresh to run.
6. Suggestion: Keep Send refresh failure notification email to me checked to ensure notification if the gateway runs into issues.
7. Click Apply.
8. Enjoy the fresh data in your awesome Power BI reports!

Let me know if there are any questions in the comments. Thank you!

That being said every once in a while, something on the "visual layer" toolbox gets my attention and recently this has been Power BI Embedded (https://azure.microsoft.com/en-us/services/power-bi-embedded/). Simply put Power BI Embedded is a service that allows users to embedded a Power BI (https://powerbi.microsoft.com/) visual report directly into a web application. This is useful due to these challenges:

1. Application developers often want/need to be focused on the core functionality of the application – not visualizing data that is often a by-product of the application or line of business process.
2. Making engaging and attractive data visuals is a daunting task for many organizations.
3. Monetizing data often brings challenges around proper costing to users – especially in a multi-tenant environment.

It's one thing to provide a service like Power BI Embedded for internal-facing end users where many times data are protected by database or data store level security via role-based authorization (RBAC) and user accounts authenticated through a central identity management system like Active Directory, LDAP or a Kerberos KDC. It is another challenge if there are applications that bake in their own authentication and authorization mechanisms while accessing the data store via a central per-application service-user account, and those end users might even be outside your organization and too numerous to make purchasing a reporting-user license cost effective. I've worked closely with a number of applications like there and there are reasons for architecting an application like this as well as reasons for not – I won't go into these here.

Role Level Security

The good news is that Power BI Embedded can make use of a feature called Role Level Security that allows you to customize the data request with predicate values and security provide each user their own returned set of data (based on existing authorization that is built into a custom multi-tenant web application). More details of Role Level Security can be found here. This is how Power BI Embedded with Role Level Security helps solve the 3 challenges above.

Allowing application developers to focus on core application functionality

Power BI Embedded allows the insertion of a Power BI report (PBIX file) via an IFRAME. This allows for easy integration with existing web applications. REST & .NET APIs are available as well as SDKs for .NET and JavaScript.

Easily Creating Engaging & Powerful Visuals

Power BI is widely known for making it easy to create powerful & dynamic data visualizations. It provides the capability to build custom visualizations, publish to internal organization users and connect to a wide variety of static and real-time data sources ranging from files, databases, NoSQL stores, and streaming data tools.

Enabling Monetization of Data

Power BI Embedded is an Azure service and as such you can easily create multiple Power BI Embedded Workspace Collections within an Azure subscription. Within a Power BI Embedded Workspace Collections there are Workspaces and within Workspaces there are published reports. Many reports can be published to a Workspace and many Workspaces can be contained in a Collection. This hierarchy allows an organization to know which Collections, Workspaces and Reports are being used. Power BI Embedded billing does not depend on a user or application count but instead is per session, more details on per session billing can be found here (as well as the latest prices).

Additional resources for Power BI Embedded & Role Level Security to get started:

• What is Power Bi Embedded: https://azure.microsoft.com/en-us/documentation/articles/power-bi-embedded-what-is-power-bi-embedded/
• Common Power BI Embedded scenarios: https://azure.microsoft.com/en-us/documentation/articles/power-bi-embedded-scenarios/
• TechNet Virtual Lab: Developing a Web App with Azure Power BI Embedded: https://vlabs.holsystems.com/vlabs/technet?eng=VLabs&auth=none&src=vlabs&altadd=true&labid=25796&lod=true
• Get Started with Power BI Embedded sample app: https://azure.microsoft.com/en-us/documentation/articles/power-bi-embedded-get-started-sample/