I’d been using Visual Studio LightSwtich for this, mostly the VB’s SilverLight project types.

During the last few months, I had to take these skills up several notches, in order to develop a couple more complex applications for my team. The first still focused on the Windows Client thru SilverLight on a 2 Tier deployment. The last one had stringent requirements, including having to run on other browsers other than IE (so, no SilverLight), and enable users from outside the company (not in our AD domain)… so I jumped to learn about the Visual Studio LightSwitch world of HTML5 projects on a 3 Tier deployment… Enter JavaScript & CSS & DOM, asynch, promises, etc…

So after going thru all of LightSwitch's documentation, tutorials, blogs, the help website, community forum, and many others… I started researching the more advanced challenges. And found great & lengthy discussions around a few common use cases, such as creating records on a table thru code, and get in the UI information about the current user; which were very informative, but didn’t totally solve the problems I was particularly facing. So let me share here what I learnt.

Requirements

These were the key design requirements for the solution:

1. Backend on Azure SQL DB.
2. Deployment on an Azure Web Site.
3. Support for Modern Browsers.
4. Users may be from outside the company.
5. Track usage of the app.
6. Users can only see what they own (per role).

So these led me to decide for HTML5 (Req. #2, #3, #4), 3 Tier Deployment (Req. #1, #2, #3), using Forms Authentication (Req. #1, #2, #4). To reuse my previous experience, the server code would be on VB.

Track Usage & Getting the User’s login

I see Req. #5 as a common need for having some instrumentation to know basic things on usage of the app. Req. #6 drove the need to know a lot more about the logged user than just it’s login alias/name.

This depicts kind of what I needed:

1. Get the user’s name.
2. Get the user’s login.
3. Get the user’s role(s) in the app.
4. If the role is “Partner”, then get the Company Name.
5. Get appropriate counts from across related tables filtered for this Company.

So, I created a screen for Home page of type “Browse” with no data associated to it.

• Added a Local Property (thru the “Add Data Item” button in the screen designer), of type String named “UserInfoString” that will show the welcome string message top right.
• Added Local Properties of type Integer for the counts (named “CntLocations” and “CntOffers”).
• Dropped these properties into the designer, at the top:
  
• For example, the UserInfoString is formatted to not show the name and right aligned (there’s a bit of CSS applied to it for color & format):
  

Now, to get the appropriate values into these Local Properties, I had to write code in the “created” method for the screen:

1. Open a connection to the data.
2. A pointer “varTrackThisVisit” to a new record on the “AppUsersSet” table in the “ApplicationData” data source (under the Server project of the solution). [This is the table where I track screen’s usage (instrumentation), and while doing that, I collect info about the user; see below.]
3. The “Screen” field of this new record (referenced by “varTrackThisVisit.Screen”) receives the DisplayName of this screen (in this case is the Home screen, but you can use this code in other screens).
4. After passing the screen’s name, now I save this new record thru the connection opened in step 1:
   1. Explained below: when I save the record, I take the opportunity that the “Inserting” method runs in the server, to collect info about the user.
   2. I use the “then()” promise to ensure the data gets saved before continuing the processing for 5, 6, 7.
5. Once the record finished saving (thanks to the “then”), I start concatenating the string into the Local Property for the user info.
6. If the user has a role (checking that the role is NOT null or empty, by going thru the ELSE of the IF statement), I add the role portion of the string (by using the “+=”).
7. If the role of the user is “Partner”, and the company name is not null (the UserPartner field), then I finish concatenating the company’s name to the string.
8. I get the counts of Locations and Offers into the other two Local Properties created earlier in the screen.
   1. Explained below: to get the appropriate count of records for the user logged in, I’ve a FILTER condition in these tables.

That will give me what I needed to show in the screen. Before I cover how I get the User’s info, let me show the CSS code I used for the formatting of the “UserInfoString” Local Property in the Home screen, added as part of the “PostRender” method:

Now, the “AppUsers” table is defined as:

On the “Inserting” method of this table:

1. When you’re saving a new record (from the “saveChanges()” in the earlier code) the “Inserting” method launches. The “entity” refers to this new record, so in this step we get the PersonID (login name) and FullName from the User of the Application (as defined when users are created for the Forms Authentication (the SilverLight project to manage the auth)). [You could also use User.Name instead of User.PersonID].
2. Here we need to concatenate the possible various roles a user may have.
3. So, we go thru all the roles.
4. Concatenating into the ThisUserRoles variable.
5. And now we assign the roles to the UserRole field of this new record.
6. Here we get the Company’s name for this user. There’s a table in the model named “PartnersSet” that have info about the company, as well as the user login for this company. The PartnersSet table has a FILTER for the current user (see below).
7. So, if there’s any company for this user:
8. We assign the company’s name to the UserPartner field of this new record. [Have to use “first()” because it’s a set of records, however it’ll have only one record.]

So, the FILTER for the PartnersSet table is:

1. The Partners table with some of its fields shown, related to the company.
2. Get to the FILTER to enter the code.
3. The field “SecurityLogin” will relate to the user’s login from the Forms Authentication entry (the security/app administration needs to link these with the same login, in the screen that that maintains this table).

1. Will filter the records to only the ones this user can see, but only if this user is a Partner (by having the “MSPartners” permission). If not, then no filter to the table gets applied, so administrators can access all records).
2. The field “SecurityLogin” is used to filter by the User.Name logged in the app.

Similar filters were applied to the related tables “PartnersLocationsSet” and “PartnersOffersSet”, so we can have the appropriate count of records. But as these are related tables to Partners, the field used in each table needs to navigate thru the relationships:

1. The “PartnersLocationsSet” table relates directly to the “Partners” table, so the navigation to the “SecurityLogin” field in order to apply the filter has one hop.
2. The “PartnersOffersSet” table relates to the “Partners” table thru the “PartnersLocations” table, so the navigation to the “SecurityLogin” field in order to apply the filter has two hops.

This covers the objective I had. From this foundation, a lot more now can be done to extend functionality. Hope this helps.

The majority of BI-related activities require measuring something on your business process. These measures usually relate to one of:

1. Inputs to a business process (usually leading indicators, i.e. number of incidents occurring; volume of calls happening; duration of calls; number of transactions).
2. How the process performs (i.e. number of tickets per analyst; time to resolution; length of queues).
3. Outputs of that process (usually lagging indicators, i.e. satisfaction scores; TTR [Avg. Time To Resolution]; costs; throughput; sales).

In almost all of these cases we need data over time to reflect how these measures progress, and be able to make decisions to drive improvements and also verify the impact of previous decisions (i.e. did adding a resource help improve satisfaction scores or TTR?).

One of the most common analyses in these cases relates to measuring the frequencies of these events happening (i.e. how often?), in order to understand the most common cases, as well as the least common ones, and ultimately the probability of a value to happen. So, counting how often the transactions happen on intervals (AKA ranges or bands) is very useful information.

Let’s take a very simple example, in which our dataset is Transaction IDs and its time to resolution in minutes (Duration):

HowTo

If we use the articles referenced at the bottom of this post, we could use those techniques to define the specific ranges we want; however, that assumes we know the data (meaning that we know min/max values and possible ranges). If we want to do a more generic analysis, we’ll have to calculate dynamically the ranges based on the input data, and the number of ranges in which we want to analyze it.

Let’s say we want to know the frequencies of these durations on intervals of 2 minutes, and see the distribution.

For that, we need to divide the range of values in equal intervals of 2 minutes. Then, we’ll count how many transactions happened on each interval. When we picture that with a bar chart with the ranges as X axis, and the count of transactions as Y, we’ll obtain the empirical distribution of that data set.

We’ll need these 4 elements:

• A measure to obtain the minimum duration across all the dataset.
• A column that calculates the range (as a number) to which the current Duration for a given transaction belongs to.
• A second column that calculates the label for the range (so we use as X axis in the chart).
• A measure that calculates the number of transactions on each range.

The first measure calculates the minimum duration of the entire dataset:

With that, we can calculate a column for the range to which this transaction’s duration belongs to (this way we calculate once the MIN(Duration) and use multiple times):

Then we calculate another column for the label of the range, which we’ll use in the chart:

And finally, we calculate the measure for the frequency (i.e. how many transactions had a duration within a given range?):

Results

We can now create a bar chart with the Range Label in the X axis, and Frequency on the Y axis:

which reflects the real distribution of durations in that dataset (which I call here “empirical distribution” because it’s based on the data itself and not on any statistical distribution)… the “shape” of the bar chart reflects the actual distribution of the dataset. As over time you refresh the dataset, the ranges will recalculate and reflect the new buckets:

This information can be used to calculate the empirical probability of new values on these ranges, if conditions in the business process don’t change. [I call it “empirical” because, as mentioned above, we use the actual data and not any theoretical or statistical distribution or any kind of sampling.]  If you make a decision in your business process (i.e. adding 2 people to help resolve tickets faster), then a change in the “shape” of your KPI (“Duration” in this case) would reflect the impact of this decision on the business process, and thus give you a way to assess how valuable that decision was (a topic for a future post).

Related Posts

• “Ranges as Filters for Dates”.
• “Grouping in Ranges/Bands”.

In my previous post “Ranges as Filters for Dates” I covered a way to group on ranges of months that we could use in selections. In this post I’ll share a way to simplify the measures using DAX’s SWITCH function, and also apply it to other cases.

HowTo

In the previous post we used several nested IF statements to calculate the column “Time Bucket”… let’s simplify that by using the SWITCH function.

• The first argument is the expression to evaluate. Because we’re creating time buckets, we need conditions that will evaluate to TRUE/FALSE, and we want to take as value the first TRUE statement… so, the expression is TRUE().
• Then we have the list of pairs of “condition to evaluate”/”value to return if true” in order of precedence, because the SWITCH function will return the value of the first condition that finds being TRUE.
• Lastly, the value for the ELSE.

Similarly, we can use SWITCH to simplify the column’s formula that will return a numeric value that we’ll use to order the previous column as explained in that post.

Results

Same as in the previous post. For example:

Related Posts

• “Ranges as Filters for Dates” as mentioned above.
• “Use Ranges/Bands for measuring Frequencies and visualizing Empirical Distributions”.

We may have data across months, and we’d like to, at times, focus on the next 3 months, the following quarter, or the first 6 months… or similar, i.e. how many people between 20-30 years old, or 30-40, etc. How do we define these ranges, and how can we visualize them?

HowTo

Let’s say we have a dataset measuring costs for projects across two years. We’d like to have ranges for:

• All months before today.
• This current month.
• The next 3 months.
• The following 3 months.
• The next year.

These calculations will be needed in the “Dates” table. First we need a column to calculate the buckets (in this case, the number of months from today):

Then, another column to calculate the actual range based on the buckets (these could be combined):

Finally, a column that we’ll use to order the ranges when we display them (on a logical order that makes sense, and not alphabetically):

When we select the “Time Bucket” column, we can use the “Sort By Column” option to choose “TimeBucketOrder”:

Now we can create visualizations like these:

Results

If we filter by “Past”, will take all months before the current one:

Current and next quarter:

And if we select America’s region, we highlight the values for that region on the selected range of dates:

Related Posts

• “Grouping in Ranges/Bands” for ways to simplify a bit this scenario.
• “Use Ranges/Bands for measuring Frequencies and visualizing Empirical Distributions”.

If we increase the current forecast by 5%, what would have happened with the variance to forecast? If we had reduced by 3% instead, then what? How can we provide an input parameter to calculate variables and see the impact of that change in the charts and trends?

Let’s start from a simple report that shows Actual and Forecast in Euros for the year, and its Variance in EU and %. The first column shows the current values, the second column in blue shows our estimations after we input a parameter of change (they are the same at start). The Waterfall chart shows the cumulative variance per region. The bars shows actuals, and the lines will show current (black) versus estimated (red) forecast (they overlap at start):

There’s also a filter at the page level for our input parameter, which starts at 0:

So, if we enter a 7.5 value on that parameter, the Estimated Forecast will increase by 7.5%, and this would have been the hypothetical impact:

• Total variance to forecast would have been 11.6% or almost 15MM EU.
• The red line of the estimated forecast now departs from the black one.

HowTo

We start with the measures in green on the first column, then we need to implement the selection of the parameter, and finally the estimated measures using the chosen value.

1. The current measures

We start with these four measures on the “Plans” table: Actuals, Forecast, and Variance in Euros, and also Variance in %:

2. The parameter

Power BI Desktop doesn’t support parameters yet, so how can we simulate it for our specific purpose? What if we had a list of values to chose from? So then we could filter that list with our selection, and we can use that filtered value in our calculations.

We can use a couple features of Power BI Desktop that allows us to create a table by entering our own data (manually or copy/paste from i.e. Excel). This feature is called “Enter Data” in the “External Data” group on the “Home” ribbon:

Which opens the “Create Table” dialog, in which you can name the table, create columns, and enter data for it (or copy/paste from i.e. Excel):

Let’s say we want in our model to choose What-If values in the range of –15 to 15 in 0.5 increments… this list can easily be build in Excel:

Now we select this list of values (including the label that will become the name of the field we enter on creating the table), and paste it:

The “Load” button will create the table; the “Edit” button will open the table in Query Editor in case we want to make additional modifications.

How will we use this table? We’ll add the “pForecast” field to the page filter area, where the users will be able to enter their chosen value. (We could also use this list of values as a Filter on the canvas, see example in this blog post).

3. The estimation measures

Now we can calculate the measures for “estimated” values given the selections on the parameter. These are how we define them:

• We increase (if the selection in the parameter was positive) or decrease (if negative) the previous value for Forecast, by converting the selected value of the parameter to percentage first.
• The “MIN” function would ensure that in case the user selected multiple options, only the first will be considered in subsequent calculations (this would only happen if we use this as a Filter list on the canvas, as in the blog post referenced earlier; otherwise it has no ill effect).
• An improvement to this formula could be to account for the user entering values that don’t exist in the list.

With these, we can build the visualizations… a Multi-row Card for current values:

A Multi-row Card for estimated values:

A Waterfall Chart for the regions:

And a Line and Stacked Column Chart for actuals versus the two forecasts:

Result

Now… what if we reduce the forecast (thus saving some budget) by 5%? We enter “-5” on the parameter:

• We clearly see that Asia will go out of budget by 1.3MM EU.

What if we reduce by 3%?

And if we select the bar for Europe, the cross-filtering will give us the information for Europe:

Let’s assume we have data about projects, and we want to rank the projects by budget, so that if we filter the data by some criteria (i.e. region, country, or some other classification) the ranking recalculates for the new scope. We also want some metrics calculated for the TopN projects, i.e. total budget for TopN and percent to overall budget. But most importantly, we want TopN to be a dynamic selection, meaning that in using the report, we want to focus on Top5, or Top10, or Top15, etc. by changing our selection at any time.

HowTo

Power BI Desktop supports DAX for calculations and data analysis problems. DAX (Data Analysis eXpressions) is a collection of functions, operators, and constants that can be used in formulas or expressions to calculate and return values. A couple of those that we’ll use here are RANKX and TOPN (see the full DAX Function Reference here).

So, we need to rank the projects first. Then we need to implement the option’s selection of the TopN. Then we define the metrics. And finally, we put everything together with some charts & tables using Power BI Desktop.

1. Ranking the Projects

We will rank the projects by total budget, so we need a “Total_Budget” (with an underscore) measure (not a field, as we need some aggregation), based on the “Total Budget” (without underscore) field… both of these from the “Budget” table:

Total_Budget = SUM(Budget[Total Budget])

On the table of Projects, we create a new measure by right-clicking the “Projects” table on the right-hand pane for Fields (or clicking the “…” to the right of it):

And we define the measure (named “Rank” in this case) as follows:

Rank = RANKX(ALL(Projects), Budget[Total_Budget], ,DESC)

• As this measure is on the “Projects” table, the default context for the measure will be the row for which it’s being calculated… and we don’t want the ranking to happen only for that row, but for all the Projects (hence the “ALL(Projects)” as the scope for the ranking).
• Then we define the criteria for ranking the projects; in this case, the total budget for each project that comes from the other measure in the “Budget” table indicated above, with the filtering context of these two tables being related to each other (there’s a relationship between Projects and Budget thru ProjIDs).
• And finally, the bigger the budget, the first to be ranked… so it’s in DESCending order.

Now we can use this measure in a table with other Projects’ fields, and order by it in ranking order by descending total budget.

2. Option’s Selection for TopN

We want to allow the selection of TopN, where N could be 5, 10, 15, 20, etc. So, we can use a couple features of Power BI Desktop that allows us to create a table by entering our own data (manually or copy/paste from i.e. Excel). This feature is called “Enter Data” in the “External Data” group on the “Home” ribbon:

Which opens the “Create Table” dialog, in which we can name the table, create columns, and enter data for it (or copy/paste from i.e. Excel):

The “Load” button will create the table; the “Edit” button will open the table in Query Editor in case we want to make additional modifications.

How will we use this table? We’ll add a filter visualization to the canvas, so these values will be filter options for us to choose from. From the runtime perspective, when we filter a table by a slicer, only records where the criteria is true will be in context, cross-filtering other tables related to it. In this case, there’s no relationship to other tables, but we can create a measure in this new table that will hold the value that was chosen, and then we can use it in other measures’ calculations later… so, we create the following measure on this new “TopN Options” table:

• The “MIN” function will ensure that in case the user selected multiple options, only the first will be considered in subsequent calculations.

3. Metrics for TopN

Now that we have the selection of TopN, we can create a couple measures on the “Projects”:

• This calculates the total budget for the TopN projects.
• The TOPN function takes the value for N (calculated previously from selecting it in the filter with the previous step).
• Then it takes the Table to get the TopN from, and the criteria for deciding what the TopN are.
• This function has some similarities with RANKX, in that it calculates the rank of the projects based on total budget descending, and selects the TopN (instead of just returning the rank’s value).

The second measure will calculate the percentage of this total budget for TopN projects, against the overall total budget:

Another measure we can use for displaying on tables, relates to highlighting which projects are the TopN based on current selections, by returning an arrow we can use as a field in the table for the TopN projects:

• If the rank for a given project is less or equal to the TopN value selected by the user, then an arrow will be returned for display/highlighting in a table.

4. Using all these with Charts and Tables

Here’s how we could display some of these measures and fields:

• The filter is an Slicer configured as:
  
• The two metrics are Cards configured as:
  
  
• The bars are Stacked Bar Chart configured as:
  

Then we could have a table with fields from Project and Budget and some of these new measures, ordered by the Rank column:

Which is a Table visualization configured as:

So, if we select Top15, it all gets filtered accordingly:

Taking advantage of Power BI’s cross filtering, if we select the bar for “Asia”, all other controls will get cross filtered, the ranking recalculated to this new subset of projects, and we effectively get the information for the “Top15 projects in Asia”:

A final example for the “Top10 projects in Europe”:

Which tells us that “the Top 10 projects in Europe represent 31.7% of the total 32MM Euros (the total of the table’s column for budget) across all projects there, amounting to 10.2MM Euros, where the Top3 reach about 4.8M Euros (the first three rows)”.

New meaning for BI: Business Insight

Key components of BI

• Focus on the business.
• A special structure for the data delivered to the business via several mechanisms.
• Iteratively develop the overall BI environment in manageable lifecycle increments rather than attempting a galactic Big Bang.

• “We have mountains of data in our apps, but we can’t access it.”
• “We need to slice and dice the data in many ways.”
• “You’ve got to make it easy for business people to access the data directly.”
• “Just show me what’s important.”
• “It drives me crazy to have two people present the same business metrics at a meeting, but with different numbers.”
• “We want people to use information to support more fact-based decision making.”
• “We spend way too much time in producing data for reviews.”
• “In spite that we have an official scorecard, during review sessions some offline spreadsheets come out questioning the data. From there, the review is spent second guessing numbers instead of driving business discussion.”

ETL Systems

• Removes mistakes and corrects missing data.
• Provides documented measures of confidence in data.
• Captures the flow of transactional data for safekeeping (history).
• Adjusts data from multiple sources to use it together.
• Correlates data across disparate systems to get views into the business that you could not get otherwise.
• Structures data for use by end-user tools.

Data Marts and Data Warehouses

Summary

(Partially inspired on Kimball methodology. See more at www.kimballgroup.com/.)

• Not having data to really understand your business standing,
• Not having data to evaluate if you have the capability to deliver what you think/plan as targets,
• Not having data to demonstrate if your actions have real impact, and
• Not having data to put controls in place to identify deviations.