Scott Larson's Blog

Data Visualization And the Art of Simplification

Scott_MS — Fri, 10 Nov 2017 11:32:11 GMT

"Clutter and confusion are failures of design, not attributes of information." - Edward Tufte I recently had the opportunity to speak with two groups quite different from my usual audience of experienced analysts and business intelligence professionals. The first was freshman journalism students at Stanford, and the second, also students, were quants beginning internships at one of my government customers. In both cases, I had been invited to provide an introduction to the fundamentals of data visualization to these fresh audiences. I was honored and enthusiastic about these presentations. I began to reflect on how to make the experience as beneficial for both groups of students, who would apply their knowledge of BI and data visualization in very different ways: the journalists to make their articles more impactful, the quants to influence public sector policy and decisions. I had to boil the concepts down to their essence. I spent several weeks thinking through solutions I've implemented, presentations I've given, and past customer conversations searching for common themes and threads. To me…it all comes down to this:

Know your audience. Tell a good story. Simplify accordingly.

Know your audience

A C-level executive cares about different things than a line level manager. A line level manager cares about different things than an analyst. Painfully obvious, I know that. Nonetheless, I often see dashboards that look like the author forgot that the ultimate goal is to help inform the viewer...not to show off their particular tool's vast capabilities. Also, realize that the best data visualizations are interactive, so you have to consider not only how the visualization looks when you publish it, you have to anticipate how different users will interact with it. Take for example, a dashboard intended to give a city council member a glimpse into the state of their district. Council members are very busy, may not be very technical, and probably won't attend training for data visualization tools. Perhaps a simple visualization like this is in order: For those that want more detail, you could allow them to optionally drill through to detail: This drill through could take them to a more detailed analysis of permits by zip code: This last visualization, while informative to an interested data analyst, might be TMI for the council member who simply wanted to know basic permit and 311 request levels. But, providing an intuitive path to this detail is a key element of modern data visualization design.

Tell a good story

Sometimes you know what the story is...new construction permits are up and illegal dumping is down in areas targeted with a revitalization campaign. In those cases, sometimes you just want to hit them over the head with your point...I call this "USA Today Style". (Below example is made up) In these cases, my advice is to take the advice of my high-school English teacher and Have A Clearly-Stated Thesis. Yes, I liked math better than English, but I attribute much of my moderate career success to two really good English teachers...a blog for another day. Sometimes you don't know what the story is, and you've been tasked with enabling others to find that story. Again, this is another blog for another day, but for inspiration, check out the way my colleague guides the viewer through a process. The below visualization is interactive...feel free to interact.

Simplify accordingly

I am striving to inform, not to impress. When I create data visualizations for customers, I obsess over eliminating clutter. Gridlines, titles, borders, data labels...often they are necessary to convey information. Often they are redundant and simply clutter the screen. As an example, let's take the default display for a Bar Chart in Power BI. Looks ok, right? It's easy to accept those defaults and move on. However, some simple changes will clean that up dramatically. Why require the eye to move off to the left to read the # of requests...let's eliminate the Y axis and put that data values right on those bars.

1) Click on the data visualization, and in the Visualization Pane, click on the paintbrush (format) and toggle the Y-Axis to off

2) Toggle Data labels to On and Increase the Font Size

3) If the intent of this visualization is to highlight the most common request types, perhaps you limit the visualization to Top 5.

4) Don't forget to change the title. I like to center the title and increase the font

As you can see, a few simple steps can dramatically simplify a visualization. While this isn't always the best approach, my point is to take a good look at your visualization and eliminate anything that doesn't provide additional value.

In summary

Know your audience.

Imagine yourself explaining your observations to your audience. Are they impatient? Are they asking you to drill into the next layer of data? Do they have natural curiosity about the data or do they just want a topline observation? Always remember, not everyone is excited about data as we are.

Tell a good story.

Why are YOU excited about the data? Why does your data matter to your audience? That's your thesis. Make sure that it is clear and concise. Make sure that any supporting data is easily accessible within your data visualization environment through user-interaction. Default interactions and drill-through actions in Power BI are good ways to provide this interactivity. More on interactions here. More on drill-through here.

Simplify accordingly.

Take a good look at the elements in your data visualization. Can you eliminate gridlines? Can you eliminate axis labels or values? Can you apply any "Top N" filters to eliminate elements that don't add value? Can a wise use of text help inform the audience?

Additional Reading

If I was a college professor teaching this subject, I would put the following required readings on the syllabus. The Visual Display of Quantitative Information by Tufte, Edward Story Telling With Data by Cole Nussbaumer Knaffic The Truthful Art: Data, Charts, and Maps for Communication by Alberto Cairo

Power BI and the World Bank API

Scott_MS — Tue, 14 Feb 2017 21:09:09 GMT

As promised, in this blog, I will create a Power BI visualization based on Hans Rosling's Ted Talk seen here. While you don't need any Power BI experience to follow along and recreate this, if you want a basic tutorial I would suggest you watch these short videos here first. It's not that anything I do is difficult, but the nature of the source data and my desire to add additional data sources over time, led me to the decision to spend a little extra time "modeling", or organizing and relating my data to provide a foundation for current AND future analysis. My goals in this demo are simple:

Mashup 3 different time-series indicators from the World Bank; Total Population, Fertility Rate, and Life Expectancy At Birth
Create a visualization that shows how the third world has changed over the last 50 years, in ways that you may not have expected

The World Bank API

The World Bank has a pretty good open data site at data.worldbank.org. There are some great datasets and analysis based on global development data. However, I must admit, when I was searching for specific indicators and how I would access them programmatically, it was a little frustrating to navigate. To save you some of that frustration, I've provided the URLs I use below. All Countries, By Year 1960-2014 Population, Total: http://api.worldbank.org/countries/all/indicators/SP.POP.TOTL/?format=json&date=1960:2014&per_page=20000 Life Expectancy At Birth: http://api.worldbank.org/countries/all/indicators/SP.DYN.LE00.IN/?format=json&date=1960:2014&per_page=20000 Fertility Rate, Births Per Woman: http://api.worldbank.org/countries/all/indicators/SP.DYN.TFRT.IN/?format=json&date=1960:2014&per_page=20000 World Bank Country Classifications: This will allow me to slice by various classifications: http://api.worldbank.org/countries/?format=json&per_page=400

Step 1: Getting the Data Into Power BI

People often think of Power BI as a data visualization tool. But it also has really powerful features for getting data, cleansing and transforming data, and mashing up data from multiple sources. Our first step is to get data into Power BI from a REST API. Here we will use 3 separate API calls to get Population, Fertility, and Life Expectancy data from 1960 through 2014. In this case, the data is returned in JSON format, which we will parse using the query editor. As you follow the steps below, note the Applied Steps section on the right. Power BI is keeping track of each transformation/change as you make it, so the next time you refresh this data it will reapply these changes.

On the top ribbon, navigate to Get Data=>Web

2. In the From Web dialog box, paste in the url http://api.worldbank.org/countries/all/indicators/SP.POP.TOTL/?format=json&date=1960:2014&per_page=20000

3. Click On List

4. Click on To Table

5. Expand Arrows, uncheck "use original column name", click OK

6. Expand the country column, uncheck Use original column name as prefix

7. Rename columns

id = Country ID value.1 = Country value = Population

8. Select your Population column, select Data Type=>Whole Number

9. Select the Date column, select Data Type=>Date

10. Right click on the decimal field and remove it. Also remove the indicator ID field

11. Rename the query to Population, and click Close & Apply

12. Repeat this process for http://api.worldbank.org/countries/all/indicators/SP.DYN.LE00.IN/?format=json&date=1960:2014&per_page=20000

a) When you rename the columns, rename them as follows

id= Country Code

value.1 = Country value = Life Expectancy

b) Change the Life Expectancy Data Type to Decimal

c) Name the query Life Expectancy and click Close and Apply

12. Repeat this process for http://api.worldbank.org/countries/all/indicators/SP.DYN.TFRT.IN/?format=json&date=1960:2014&per_page=20000

a) When you rename the columns, rename them as follows

id = Country Code value.1 = Country value = Fertility Rate

b) Change the Fertility Rate Data Type to Decimal

c) Name the query Fertility Rate and click Close and Apply

Step 2: Merge the Data Sets

We now have three tables; population, life expectancy, and fertility rate. But logically, each table represents a single "attribute" of a country for a given year, so I thought it would make sense to merge them into a single table. This will make it easier for end users to understand.

On the top ribbon select Edit Queries
On the top ribbon of the Edit Query screen select Merge Queries=>Merge Queries As New

3. Each of these tables has Country Code, Country Name, and date in common. Merge them into one by selecting the Population table from the first dropdown and the Fertility Rate on the second drop down. Then in each table, while holding Ctrl down, select Country Code, Country Name, and date in each table. You should see that it has matched 14520 out of 14520 records.

You can see 14520 out of 14520 fields are matched.

4. Click OK

5. The Fertility Rate table can now be expanded out. Click on the arrows by New Column, uncheck use original column name as prefix and select the Fertility Rate column

6. Click on the Merge1 Query on the left Query Pane. Select Merge Queries again. This time, instead of selecting Merge Queries as New, Select Merge Queries

7. The first table will already be set to Merge1. Choose the Life Expectancy table for the second drop down. On each table, select County Code, Country Name, and date.

8. Click OK and Expand the new column like you did in step 5 of this section. Uncheck use original column name as prefix and select Life Expectancy

9. Name the new merged table World Development Indicators

10. Select Close and Apply

11. Since we've merged the information from the three other tables into one table, we can hide those three tables. In the field pane, select Fertility, right click and select Hide. Repeat for Population and Life Expectancy.

We now can see one table, World Development Indicators, containing all the information we need.

Step 3: Create Scatter Plot Visualization

Select the Scatter Chart from the Visualizations Pane

2. In the visualizations pane details, drag Country Name to the Legend, Fertility Rate to the X-Axis, Life Expectancy to the Y-Axis, and date to the Play Axis.

3. Press the Play button to watch the neat animation.

Step 4: Add World Bank Classifications table to allow grouping and filtering by country characteristics

The world bank categorizes countries based on region, lending type, and income level. We want to expand our visualization to let users filter by these additional attributes. While we could use merge the data set like we did in step 2, I wanted to keep the classification table as a completely separate table...for those of your familiar with dimensional modeling, it is essentially serving as a "dimension" table related to our indicator "fact" table. For those of you not familiar with dimensional modeling, forget I said that.

1. In the Power BI Ribbon, click Get Data=>Web and paste this URL: http://api.worldbank.org/countries/?format=json&per_page=400

2. Follow the steps listed in Step 1. (Click on List, To Table, Expand the columns, uncheck Use original column name as prefix)

3. Expand region, only select value

4. Right click on the expanded column and rename column to Region

5. Expand adminregion, select value, rename column to Admin Region

6. Expand incomeLevel, select value, rename column to Income Level

7. Expand lendingType, select value, rename column to Lending Type

8. Rename query to Country Classification, click Close and Apply

Step 5: Create a relationship between the tables

One the most powerful features of Power BI is the ability to relate two or more tables together. Many of you may have done this in Access, or perhaps you are using V-lookups in excel to do something similar. Stop. Use Power BI instead. Seriously.

1. On the left hand side, click on the Relationships icon. You'll see all the tables we've created in this exercise, but the ones we've hidden will be grayed out.

2. The two tables are related by the Country Code column in the World Development Indicators table and the iso2Code in the Country Classification table. Drag Country Code from the World Development Indicators table to the iso2Code in the Country Classification table. This allows us to "lookup" additional attributes of a country, like region, income level, and lending type.

Step 6: Add Classification Filters to your page

1. On the left pane, click back to your visualization.

2. Click anywhere on your page (except on the existing scatter chart visualization), and select Region from the Country Classification table in the fields pane

3. Select the Slicer visualization

4. On the slicer you just created, click on List, and select Dropdown. Resize the filter so it takes up less space.

5. Repeat steps 2,3, and 4 but this time select the Income Level column

6. Repeat steps 2, 3, and 4 but this time select the Lending Type column

7. Arrange the filters and resize the scatter chart to taste.

So that's it. You can now impress your friends and family with your Power BI Story Telling abilities. There are many more World Development Indicators you can get from the World Bank open data site using the same process I showed in this blog. You can get a list of them here. Also, just for fun, I made a little Power BI visualization that lets you choose indicators on the left and see the REST APIs on the right. Enjoy.

Data Visualization Heroes

Scott_MS — Sat, 21 Jan 2017 10:26:53 GMT

Everybody has a hero. Your hero might be a professional athlete that you admire for their physical prowess. Your hero could be a musician that you admire for their songwriting and musical talent. To some, heroes are historical figures that discovered new lands or scientific truths, or built new machines to benefit humankind. Less often however, heroes are Swedish medical doctors, statisticians, or data visualization thought-leaders. So perhaps I am unique (or not?), that I place Hans Rosling, Edward Tufte, and Nate Silver on the same list as Nolan Ryan, Brett Favre, Peter Gabriel, and Benjamin Franklin. Sure, they didn't throw a baseball 100 miles an hour, or throw over 500 touchdowns...no platinum albums or kites in thunderstorms. But these individuals all have one belief in common, that's near and dear to my heart: The ability to tell simple, memorable, and insightful stories about your data is of paramount importance, and thoughtful data visualizations are the most powerful tool you can use to tell your story. Intuitively, I've understood this principle during my 15 year business intelligence career, but it really crystalized about 5 years ago when I saw this 20 minute Ted Talk from Hans Rosling. If you're short on time, start at 4:00. If you're not short on time, watch the whole thing. I retained more from that short presentation about social change in the world, the aids crisis in Africa, and the distribution of wealth in the world than I would have from an entire college course...all due to professor Rosling's ability to tell a story, with a clearly-stated thesis, and clear, concise visualizations reinforcing that story. I want my customers to Be Like Hans. As a Microsoft employee, when I am demonstrating some of the features of Microsoft's flagship data visualization product, Power BI, I often do a similar visualization using data from the World Bank open data site. I created this in Power BI Desktop, published to my powerbi.com account, then "published to web" so I could embed it in an iframe below. In my next blog, I'll step through how to use Power BI Desktop create this visualization using the World Bank API.