Pro Tip: In the Query tab, you can change whether the data loads to the data model or not by unchecking the 'enable load' option on the right-click menu for a query. It's super useful, give it a try.

Table.AddIndexColumn(#"Expanded Column1", "Index", 0, 1) ColumnIndex = Number.Mod([Index], List.Count(ColumnsList)) Column Name = ColumnsList{[ColumnIndex]}

Date = if([Attribute] = "DATE") then [Value] else null Value.1 =  if([Attribute] = "VALUE") then  [Value] else null Fill Down on Date Filter out nulls on Value.1

GrandTotal = CALCULATE(SUM('Demo Data'[Value]), ALL('Demo Data'[Category]))

I also did some work to build an excel workbook that translates the raw table of data provided for the competition into something more usable. If you'd like to use the workbook, I've provided a link below where you can download it. To use it, you'll either need Excel 2013 with Power Query or Power BI Desktop (free) to pull in the data, or other tools that can read Excel files.

You can see it more interactively on Docs.com here.

The treemap above shows a count of the occurrence of words in the contest data set. It's interesting to see what words are used to see what concepts or ideas are really important to the people making comments. To do that I used Power Query with various unpivot and split transforms to get to a table of words.  I then asked Power BI Desktop to plot the count of words a a treemap.  If you look at the excel file I mention below, you'll have tables of distinct words, distinct comments, and distinct documents.  So you can take this approach much further.  I'm toying with some basic heuristics to find 'meaningful' words for example and that's looking promising. 

Files on OneDrive, you'll want to grab a copy of [Master Data] WHS Master Consultation Data Collection (23 July 2015).xlsx at least. 

If I end up making a contest submission, I'll post my entire submission, including the Power BI Desktop file as well so you can play with it.

HTH,

Lukasz

This treasure trove of information invites a lot of visualizations.  But where should I begin?  

Glad you asked!  Turns out there's a catalog of all the data feeds available from the City of London in a handy CSV format. The data set has a really good set of attributes including direct links to datasets, attributions, etc.  While there is a search feature on the website, it's somehow not as interactive as I'd like.  So let's take a look at how we can sift through London's datasets using Power BI Desktop.  

Resources

Download:  London Data Feeds Index.pbix

Download:  Power BI Desktop if you don't already have it

Download:  the London Datastore Catalogue CSV file     

More data, updated more often

I started by looking just to see how many data sets London publishes and how that has changed over time.  Here's a great view that shows London is publishing or updating more data sets today than a year ago.  You can see there are 602 datasets we could analyze - a cornucopia of data!  You can see that the number of data sets updated in 2015 is already on pace to exceed 2014.  So not only is London publishing data, but it's keeping the data updated as the months go by.

Loading the data

Loading the CSV file is very easy in Power BI Desktop.  I just clicked Get Data and selected CSV.  The basic CSV was loaded correctly with all the data types correctly detected.  There were a few things I spotted that I wanted to do to make the data more useful.  To start there's a URL field - I want it to be clickable when used in the report so I changed the Data Category for the URL field to "Web URL".   Then there's the tag field.  Its a single comma separated field and I wanted to see which were the popular tags.  To do this, I needed first to isolate each tag value and then ensure I could slice and dice the data feeds based on the tags.  Getting there required two additional tables.  The first provided the unique tag values. The second related the tags to the IDs.  You'll see in the Power BI Desktop file, I created relationships across the detail table and the two new tables to ensure I could filter the values. Importantly, I used the ID column from the IdToTagsMap table in the main table for tags to force the use of the relationships during filtering.  You can see why this is needed in the image below, where the IdToTagsMap table is on the 'many' side of the relationships from the Datasets and Tags tables. 

A searchable index of London datasets

When you have so many datasets, it's hard to find all of them. So the sheet below shows how I created an interactive index of all the datasets published for London.  You can filter on Topics, by Tag, or the update frequency.  Since it's an interactive sheet, you can use any of the Fields to build additional filters or modify the visualizations.  One thing I immediately noticed is there's a ton of 'tags'.  Many datasets have unique tag values that actually are compound name that include other tags in a single long string.  So I filtered out the tags with few datasets.  I set the filter on the tags visualization so I could show all the datasets in the other charts.  This single view would make any student doing homework and looking for data from London really happy. 

Drilling into Topics and Publishers

The view below shows the top topics by number of datasets.  You can clearly see where the UK government and the City of London are spending their 'data dollars' err.. 'data pounds' :).  It's clear to see that normal topics like demographics, employment and skills, environment, transport are of primary importance.  But the 'transparency' topic is an interesting find.  Should be a useful area for citizens to dive into.   You can also see that the 3 biggest publishers, by number of datasets, are the Greater London Authority (GLA), Office of National Statistics (ONS), and Transport for London (TfL). 

Slicing and Dicing 

I built the view below to help me understand the datasets more.  I can clearly see which data sets apply to which geographic area - I especially liked the 'smallest geography' attribute since it makes finding relevant data very easy. The 'bounding box' attribute was similar so you can understand the applicability of the dataset at a glance.   Lastly, the 'date from' and 'date to' attributes help you find interesting historical reference data.  For example, one of the data sets, Global City Population Estimates,  goes all the way back to January of 1950!  It might be the subject of a future blog post.

Wrapping up

Building the initial index of London datasets was accomplished in a couple of minutes.  I was able to slice and dice quickly.  Once I got into features like analyzing by tag I had to put a little more work into it, mostly because I don't do that kind of thing very often.  It was simple to do using the built-in UI.  Having this index, that is easily accessible, searchable, and filterable with drill through to each of the pages hosting the data feeds is remarkably useful I think.  Would really like to hear from other folks who have compiled data sets like this one. 

HTH, 

-Lukasz

This post show how to show the latest date.  You can read on below, or grab the data and MaxDateExample.pbix files to see it in action yourself.  If you don't have Power BI Desktop yet, you can get it here.

Step 1: Import your data set.

In this case I have a simple table called Table1 with a column called Date.  Just click Get Data and select the excel workbooks (Book1.xlsx).  Select Table1 and press the Load button. The data will load and you'll see fields at right on the screen; you can drag them to the canvas to create a table or chart.

Step 2: Create a measure to return the Latest Date

Select the table in the fields list and press the 'New Measure' button in the ribbon. Here's the formula to use:

MaxDate = CALCULATE(MAX(Table1[Date]), ALL(Table1))

I wanted to mention why I'm using CALCULATE.  In Power BI, every time you use data, it is implicitly grouped by the categories that are used.  A measure when used in a group context, computes its value only within that group by default.  This makes sense when trying to make measures that work at any level of grouping/hierarchy.  In my "latest' case, I want to get the latest value regardless of grouping.  To do that I'm using the CALCULATE (... , ALL(...)) pattern.  It's a little confusing since the ALL(...) is called a 'filter' but it's working on grouping... yikes. You can think of a group as a kind of filter that select only rows that match the group label... so it all works out.

Step 3: Add a column to the table that tells you which rows have the latest date

Select the table in the files list and press the "New Column" button in the ribbon.  Here's the formula to use:

Is Latest = if(Table1[Date] = Table1[MaxDate], "Latest", "")

This checks if the value of the "Date" column in each row exactly matches the MaxDate value computed for all rows and writes the value "Latest" if it does.  Note that if you're using date time, this includes hours, minutes, seconds in the calculation; if you just want to know if it's on a specific day, then you'd adjust the formula.

Step 4: Use the "Is Latest" column in filters or slicers

You can now drag the "Is Latest" column to the filter area at the visualization or page level to show the value you want.

As a bonus, Power BI now support adding date and text measures to the canvas, so you can even put MaxDate in your report without jumping through hoops.  This is useful if you'd like to show the "last updated time" for your queries.

HTH,

-Lukasz

Folks,

Spread the word.  Dawn, our amazing event coordinator, has put together an 'A' list of quick demos to help you get up to speed on Power BI.  Here's the list of presenters so far.  Please tweet, share, repeat, tell your friends, and be sure to stop by to see these great sessions.   

You'll find us in front of the large touch screen PPI under the blue Cloud sign.  Can't wait!

See you there.  Of course I'll feature my talk in more depth right here on my blog as well.

-Lukasz

The week after next, I’ll be in Chicago to present at Microsoft’s Ignite conference.  I thought it would be fun to look at some data for the City of Chicago to see what I could do with it.  

Revenue is important to any city.  Developing enduring sources of revenue, whether through direct means like leasing city owned property or other ingenious ideas, is something every city planner needs to consider when thinking about where to spend precious resources.  Let’s see how the City of Chicago Data Portal (https://data.cityofchicago.org/) helps us.  I’ll use Microsoft’s Power BI Designer to do the analysis (free download).

Before I get too far into it, a word of caution: While the data and tools used are real, without more analysis no conclusions should be drawn as to what actions to take.  I’m not sure what the facts on the ground are in Chicago or any of the neighborhoods mentioned below.  Maybe someone’s already thinking about all of this.  In any case, let’s see what the numbers say…

I opened the City of Chicago Data Portal and found the “City-Owned Land Inventory” data set. 

As with any Socrata based data portals, you can get an OData url as shown below that makes it really easy to get the data into Power BI Designer.  Power BI Designer natively supports OData input so the data loads quickly and easily.

Now to do the mashup, I needed to get a sense of the rental rates in the City of Chicago.  So I looked around the web and found http://www.loopnet.com.  They have a set of tables that show various real estate market trends in Chicago, including lease rates.  There are a lot of different trends to choose from. I somewhat arbitrarily chose the Office Property Asking Rent – Lease Trends and chose the “Metro” rate for the analysis.  I wanted to use a nice representative number.

To get this data into Power BI Designer, I used the option Home->Get Data-> Web and entered the URL “http://www.loopnet.com/Chicago_Illinois_Market-Trends”.  I picker that opens has a bunch of tables in it and allows me to preview tables.  Some guess and check iteration is required, but eventually I found Table 28 is the one I wanted.

I called this table “RentalRates”. Now that I had reasonable rental rates, I could really get going with my mashup.  To figure out the “Potential Revenue” for each property I needed to multiply the square feet of the property by the rental rate.  This required some Power Query know how. 

I added a custom column called “Potential Revenue” to the “City Owned Properties” query.   The column does a look up in the RentalRates table for the rental rate for “Metro” which is what I wanted to use. 

Some technical details…

If you don’t care about technical details, skip down to the pictures below! But this little bit of exposition shows a couple of Power Query tips and tricks you might want to have in your tool belt.

Here’s what the “Potential Revenue” column’s formula looks like:

[sq_ft] *
Table.SelectColumns(
    Table.SelectRows(RentalRates, each ([Region] = "-Metro"))
    , {"$/sqft (Mar 15)"}
){0}[#"$/sqft (Mar 15)"]

Let’s step through what this does.  You need to read it from the inside out.

     Table.SelectRows(RentalRates, each ([Region] = "-Metro"))

This selects a row in the “RentalRates” table where the column “Region” has the text “-Metro” in it.  You might ask why I chose that string, it’s by inspection of the data value returned from the web page.  I could have cleaned it a little, but that’s for next time.

Next we have the Table.SelectColumns that selects the column I want.  That column as the name “$/sqft (Mar 15)”

Table.SelectColumns(
    Table.SelectRows(RentalRates, each ([Region] = "-Metro"))
    , {"$/sqft (Mar 15)"}
)

Now comes the tricky part.  What the above returns is a table with one column name “$/sqft (Mar 15)” with one row, and in that row is the value I want. So I can append the power query row indexer {0} to get the first row of the table and specify the power query column indexer [#"$/sqft (Mar 15)"] to get the value of the column named “$/sqft (Mar 15)”.  Notice the # sign.  What that does is it makes string an object that the system can compare against the column.

{0}[#"$/sqft (Mar 15)"]

Whew – so am I really crazy smart at Power Query to get this done? No not really.  The way I build that was using a trick.  In the queries list, I right clicked the RentalRates query and selected “reference”.  This creates a “RentalRates (2)” query that I can use the Power Query UI to filter to the right row and select the right column. Then I could select the cell I wanted, right click and select “Drill Down” this gives the following query.  Btw, on the “View” tab you can use the “Advanced Editor” to see and edit the queries.

let
    Source = RentalRates,
    #"Filtered Rows" = Table.SelectRows(Source, each ([Region] = "-Metro")),
    #"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"$/sqft (Mar 15)"}),
    #"$/sqft (Mar 15)" = #"Removed Other Columns"{0}[#"$/sqft (Mar 15)"]
in
    #"$/sqft (Mar 15)"

With this query, I can easily get to the expression I used.  Now I did it that way because I was curious about indexing in in Power Query (I need to know all that for my day job). 

There’s an easier way to do it, that probably few people know about!  For analysts, instead of using “Drill Down”, I probably would recommend selecting “Add as New Query” on the drop down.  This makes a new query that returns the value of the cell you selected.  By default this gives a query with the name of the column.  After renaming the query to something like “metroRentalRate”.  What you should know is that in Power Query you can referenced other queries in your formulas.  In this case it’s easy since this query returns a single value. So your final formula for the “Potential Revenue” column would look like this:

[sq_ft] * metroRentalRate

Now that’s pretty nice.  You can remove the “RentalRates(2)” query since it’s not needed without breaking anything and you’re done with that part! As a side note when doing the above, I would still use the ‘reference query’ step since it results in a more efficient set of queries – the data will be downloaded by Power Query only once, rather than twice.

Beyond the above, I did some standard tweaks to the data returned from the OData feed.  To map addresses better, I created a couple of columns to help Bing geocode the addresses better. Here are the queries:

Street Address

= [address] & ", Chicago, IL, USA"

CMTY

= [cmty_area_name] & ", Chicago, IL, USA"

I think the second one is interesting.  The cmt_area_name contains names like “Englewood” that are regions within the city.  Bing has some knowledge of geographic boundaries alike Countries, States, Cities, Zipcodes.  You mileage may vary, but it’s pretty cool when it works, which it did in this case – see the filled map below.

I also used the “replace values” option on the “vacant” column to change the values from ‘true’, ‘false’, and ‘null’ to “vacant”, “occupied”, and “not specified”.  That’s much easier on the eyes.

Show me the data

Now that all the data modeling and queries are done, I created a few pages in Power BI Designer to show some of the data.  Using a TreeMap, I’m able to see part to whole analysis.  I created the view below to show potential revenue by ward number.  As you can see 4 wards account for the bulk of the potential revenue.  You’ll also see the distribution of properties across whether they’re vacant, occupied, or not specified. One very interesting thing in the TreeMap is that you can see how the number of properties in not specified is actually quite high but the potential revenue for those properties is extremely low.  The reason is that there’s very little square footage associated with these properties.  So they might be vacant lots.  What’s interesting about that is that vacant lots are sometimes easier to redevelop than lots with building on them already.

Below, I used the Filled Map feature to see which of the communities have the most vacant properties.  Filled Maps are sometimes called choropleth maps. You can see that there are a few neighborhoods with higher densities of vacant city owned property.  That clustering makes it interesting as a potential way to focus resources on those areas to avoid a peanut butter effect.

Lastly we can look at the wards with the most vacant property and see where all those properties are.  Again, you can see that clustering when looking at each property.  This suggest there might be some really big opportunity in those areas of Chicago.

Hope you liked this analysis.  You can find the final Power BI Designer file here:  Chicago City Property.pbix

You can get Power BI Designer for free by click here.

Also, if you haven’t yet, you can signup for Power BI for free by heading over to www.PowerBI.com.

-Lukasz

You can make it fully dynamic by combining the technique Matt Masson describes here to create a date table and add to it some Power Pivot DAX functions to classify each date into the date range you'd like. 

Download a sample here

OneDrive personal doesn't load the Power View sheet so download and open the example in Excel on your desktop.

EDIT 7.27.2015  - I was asked for the formulas since some folks don't have Power View.  These should work in Power BI Desktop, but I've not tested them there.  Here you go:

Power Query query:

//let
//    CreateDateTable = (StartDate, EndDate) =>
let

    StartDate=#date(2015,1,1),
    EndDate=#date(Date.Year(DateTime.LocalNow()),12,31),

    //Create lists of month and day names for use later on
    MonthList = {"January", "February", "March", "April", "May", "June"

                 , "July", "August", "September", "October", "November", "December"},

    DayList = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"},

    //Find the number of days between the end date and the start date

    NumberOfDates = Duration.Days(EndDate-StartDate),

    //Generate a continuous list of dates from the start date to the end date

    DateList = List.Dates(StartDate, NumberOfDates, #duration(1, 0, 0, 0)),

    //Turn this list into a table

    TableFromList = Table.FromList(DateList, Splitter.SplitByNothing(), {"Date"}

                     , null, ExtraValues.Error),

    //Cast the single column in the table to type date

    ChangedType = Table.TransformColumnTypes(TableFromList,{{"Date", type date}}),

    //Add custom columns for day of month, month number, year

    DayOfMonth = Table.AddColumn(ChangedType, "DayOfMonth", each Date.Day([Date])),

    MonthNumber = Table.AddColumn(DayOfMonth, "MonthNumberOfYear", each Date.Month([Date])),

    Year = Table.AddColumn(MonthNumber, "Year", each Date.Year([Date])),

    DayOfWeekNumber = Table.AddColumn(Year, "DayOfWeekNumber", each Date.DayOfWeek([Date])+1),

    //Since Power Query doesn't have functions to return day or month names,

    //use the lists created earlier for this

    MonthName = Table.AddColumn(DayOfWeekNumber, "MonthName", each MonthList{[MonthNumberOfYear]-1}),

    DayName = Table.AddColumn(MonthName, "DayName", each DayList{[DayOfWeekNumber]-1}),

    WeekEnding = Table.AddColumn(DayName, "Week Ending", each Date.EndOfWeek([Date])),
    #"Changed Type" = Table.TransformColumnTypes(WeekEnding ,{{"DayOfMonth", Int64.Type}, {"MonthNumberOfYear", Int64.Type}, {"Year", Int64.Type}, {"DayOfWeekNumber", Int64.Type}}),
    #"Added Custom" = Table.AddColumn(#"Changed Type", "MonthYear", each Text.Range([MonthName], 0, 3) & "-" & Number.ToText([Year])),
    #"Added Custom1" = Table.AddColumn(#"Added Custom", "MonthYearNumber", each [Year] * 1000 + [MonthNumberOfYear]),
    #"Changed Type1" = Table.TransformColumnTypes(#"Added Custom1",{{"MonthYearNumber", Int64.Type}}),
    #"Duplicated Column" = Table.DuplicateColumn(#"Changed Type1", "Week Ending", "Copy of Week Ending"),
    #"Renamed Columns" = Table.RenameColumns(#"Duplicated Column",{{"Copy of Week Ending", "WeekEndingDate"}}),
    #"Changed Type2" = Table.TransformColumnTypes(#"Renamed Columns",{{"WeekEndingDate", type date}}),
    #"Renamed Columns1" = Table.RenameColumns(#"Changed Type2",{{"Week Ending", "WeekEnding"}})
in
    #"Renamed Columns1"
//in
//    CreateDateTable

DAX Measures
Today:=DATE(year(now()),MONTH(NOW()), DAY(NOW()))

DAX Calculated Columns
IsInCurrentWeek
=if([isCurrentYear] && WEEKNUM(NOW())=[WeekOfYearNumber],1,0)

IsInLastWeek
=if([isCurrentYear] && (WEEKNUM(NOW())-1)=[WeekOfYearNumber],1,0)

IsInCurrentYer
=if(YEAR(NOW())= [Year],1,0)

WeekOfYearNumber
=WEEKNUM([Date])

IsLast30Days
=if( AND( [Date]  >= [Today]  - 30 , [Date] <= [Today] ),1,0)

 HTH,

-Lukasz

Living in Seattle, it’s quite fun to keep track of what’s happening through data.  The Open Data initiative from the City of Seattle is a treasure trove of information that you can explore here: http://data.seattle.gov.  This is all power by Socrata, as are many other open data initiatives for Cities and organizations.  I wanted to explore that data using Power BI, specifically Power BI designer.  In a future post, I’ll show how to monitor insights using the new Power BI Dashboards feature, but that’s for later.

I wanted to look at Building Permit data – since there’s a lot of construction happening in and around town.  I was able to find the “Building Permits : Current” data set which lists “Building Permits issued in the past five years or currently in progress”.  You can see it here:

https://data.seattle.gov/Permitting/Building-Permits-Current/mags-97de

I pulled that data into Power BI Designer.  You can download the designer at the link below if you’d like to try it for yourself: 

http://www.powerbi.com/dashboards/downloads/designer/

You can follow the steps below or play with the final file, download it from here:

Seattle Building Permits OData.pbix 

Let’s walk through step by step:

The data.seattle.gov site for building permits that exposed through Socrata.
1. Click the Export option
2. Select OData
3. Copy the OData URL

Open Power BI Designer.
1. Under "New Source" select "OData Feed".

Fill in the URL and press OK.
1. Put the OData Feed URL here.

Start using Power BI Designer to explore the data.
1. There are the fields from the OData Feed.  We call this the "Fields List".
2. You can drag fields from the "Fields List" into this area we call the "Canvas".

If you drag "value" to the canvas, a default visual is created.
1. This is the field I dragged.  You can also check the check box.
2. Use this button to quickly change the visual. 
3. You can also change visuals from here. Just make sure to select the visual you want to change first.

Here's my first report after playing with the data for just a few minutes.
1. This is the total value of all permits returned by the OData feed.
2. This is the distinct count of permits by permit_type.
3. This tree map shows that Commercial is the largest category by total value.  Interestingly it shows all the applicant names and how those applicants contribute to the total value of each category. 
4. This trend shows the distinct count applications by date.
5. This data point is fascinating!  The data sets has 5 years of building permit data and any "current" permits.  So permits from before this point are really old.  Might be worth investigating more why they're still open.

The report is interactive by default.
1. I clicked on "COMMERCIAL" here and it  cross filtered all the other visuals for me.  So I can see the trend and totals just for what I selected.

When I switch to "Query" I can see how the data was loaded.
1. Clicking here switch to show the data that was loaded.   Clicking "Report" takes me back to the report.
2. I can have multiple queries in a report.  They all show up here.  To get more queries, select "New Source" in the ribbon.
3. I can work on the data to transform it, mash it up, clean it up, or what have you.  We call these actions "Steps".   The steps show up here.  You view and fix the steps.

Odata can contain nested data.  You can work with it without a problem.
1. This column has more data.  It will shows as "Record", "Table", or "List" in the UI.
2. Press this button to expand it.

This is my query after I expanded a few columns.  I selected the green column so they're easier to see.
1. Notice how I have multiple steps now.  They're sequential so I can click through them to see what the query is doing.
2. I can always edit a step by pressing here.

I created a page that shows where each application is planning to build.

Using filters and different visuals, you can quickly find insights.  Here quickly see which applicants have applied for permits.  I've captured a view that shows that Patterson has applied for many more permits than Belcher, for example.  But for both those applicants, the total of their permit values is larger than half a million dollars.

There it is.  A simple interactive view of Seattle building permits data.  The OData feed from Socarata works great but unfortunately doesn’t include zip code information.  That would have allowed me to create a density map of neighborhoods showing where investment is going. Maybe someone from the City of Seattle will read this, add the zip code and I’ll do a follow up. 

Next step will be publishing this to Power BI to monitor it.  I’ll do that in a future blog post.

Let me know what you think in the comments.  Would love to answer any questions you might have.

HTH,

Lukasz