PowerBI supports streaming dataset through an API that can be used to constantly push data to the service for real-time refresh. This post outlines the steps to accomplish it with Python script.

1. Create a StreamDataSet in PowerBI

   In your workspace, Click on "+ Create" and choose "Streaming dataset"

Once you choose "streaming dataset" option, pick "API" streaming option since we will use this API to populate data from Python

Once you click Next, enter the data set name and the values that you need to post on the API. These values will then have to be used as data set on the dashboard.

Once you click on the create button, the service will create a Push URL that will be the API to push the data to. Copy the PUSH URL to the python script.

Then create a Dashboard and Add a tile and choose 'Custom streaming data'. You will then see the streaming data sets that was created in the previous step. You can add the elements that was created as part of the steam set.

Once the dashboard Is created, copy the PUSH URL in the script below. The below script reads data from a SQL Server table at regular interval and aggregates and generates the sum of the Income value. The generated totalincome is converted to a JSON format required for the Push URL and then post it to the API.

The script runs in an infinite loop, by reading data and pushing data into the PowerBI streaming set. In a real world scenario, the data read from the DB will have be incremental either based on date, ID or other relevant column.

In this blog post, we will be discussing how to merge a normalized data structure to a Key-Value pair with a JSON.

Let's pick a scenario with customer and their Orders

Table 1 – Customer and Order Relationship

The relationship between customer and order is 1..n. The order list below belongs to two different customers. The dataset in Table 2 is transformed and stored in the DB (either relational or NoSQL dbs) as mentioned in Table 3

Table 2 – Orders

Table 3 – Customer and Orders in key-value pair (JSON)

As customers place new orders, the dataset in Table 4 must be merged with the already existing JSON set to produce a table set as mentioned in Table 5.

Table 4 – New orders from Customer

Table 5 – Merged orders

The python script extracts from old JSON dataset and integrates new orders and rebuild the JSON.

The steps are

1. Extract the data file with the EXTRACT statement
2. REDUCERS are spun based on the customer code. Too little reducers or too many reducers may both cause performance issues. Identify a column that can fairly split, but make sure not to specify a unique column.
3. For every reduced data set, the python script is invoked with the DATA FRAME. Add another column to the data frame "sguid" and generate a new encoded UID.
4. The output produced out of the reducer will have a new column sguid

Note : Follow these instructions to enable U-SQL extensions on your ADL-A account

The script below validates each of input files in the folder and the python script splits and count the number of columns in each row of every. Those files that either have > or < than 9 columns in any of its rows are all logged as Invalid files.

The EXTRACT Statement loads all files from the relative path "/DataLoads/Input" along with the filename attribute. The entire row is picked up as a single column and the column separator is assumed to be a comma (",").

The REDUCER groups the reduction by filename and then invokes the python script with the data frame. The python script calls the validate_row for each row with the row as a string. If the number of columns in the row is <9, the result will be a negative value and the result will be a positive value if the #of columns exceed 9. In essence, the result should be 0. The REDUCER produces the name of the file and the number of columns every row.

The files that do have noofcolumns <> 0 is all routed to an InvalidFile list.

Follow the instructions here to enable Python extension on the ADL-A account

Error 17883 is reported if the Worker does not yield over 60 seconds (the trace flag –T1262 can ne used to alter this frequency

When all the schedulers encounter 17883 error or all workers are blocked on a critical resource or all workers are executing long running queiries, SQL Server reports scheduler detect Deadlock conditions with the error number 17884,

for further information on this

http://technet.microsoft.com/en-us/library/cc917684.aspx#_17883

The deadlock scenario in this case cannot be completely eliminated, however, the impact/cause can be minimized. I have explained with a complete sample scenario

Create a new database and execute the script from the attached file "deadlockschemafiles.sql". This file create two tables, one indexed view and populates data into the tables.

Step A) Create a new session (Session 1) and execute

                  begin tran

                  update invoice set amount = amount + 1 where invid = 1

Step B) Create another session (Session 2) and execute

                   begin tran

                   update customer set custname = 'Customer Change1' where custid = 2

This statement in Step B will will be blocked and if you look at the execution plan of Update Customer statement  (from the attached document QueryPlan.doc Section Query Plan A ) you will see a clustered index scan on the invoice table.

This may mean that for any customer to be updated, no other session should hold exclusive lock on any of the invoice rows at that point of time since a clustered index scan is performed on invoice table.

Now we have just seen a blocking scenario, so when does it result in a deadlock.

Execute this statement from Session 1 (created with Step A)

Deadlock.zip

 http://www.microsoft.com/Sqlserver/2005/en/us/compare-features.aspx

For SQL Server 2008 features supported by each of the edtiion

http://msdn.microsoft.com/en-us/library/cc645993.aspx

declare @transactionid nvarchar(28)

select @xactid = transaction_id from sys.dm_tran_current_transaction

select @transactionid = [transaction id] from fn_dblog(null,null) where [Xact ID] = @xactid

select * from fn_dblog(null,null) where [Transaction ID] = @transactionid

The [Xact ID] introduced in SQL Server 2008 has the transaction id for the operation LOP_BEGIN_XACT and all the transaction occuring in its context share the common transaction with the [transaction id] column

        ,object_id

        ,index_id

        ,partition_number

        ,leaf_insert_count

        ,leaf_delete_count

        ,leaf_update_count

        ,leaf_ghost_count

        ,nonleaf_insert_count

        ,nonleaf_delete_count

        ,nonleaf_update_count

        ,leaf_allocation_count

        ,nonleaf_allocation_count

        ,leaf_page_merge_count

        ,nonleaf_page_merge_count

        ,range_scan_count

        ,singleton_lookup_count

        ,forwarded_fetch_count

        ,lob_fetch_in_pages

        ,lob_fetch_in_bytes

        ,lob_orphan_create_count

        ,lob_orphan_insert_count

        ,row_overflow_fetch_in_pages

        ,row_overflow_fetch_in_bytes

        ,column_value_push_off_row_count

        ,column_value_pull_in_row_count

        ,row_lock_count

        ,row_lock_wait_count

        ,row_lock_wait_in_ms

        ,page_lock_count

        ,page_lock_wait_count

        ,page_lock_wait_in_ms

        ,index_lock_promotion_attempt_count

        ,index_lock_promotion_count

        ,page_latch_wait_count

        ,page_latch_wait_in_ms

        ,page_io_latch_wait_count

        ,page_io_latch_wait_in_ms

        ,@now

from sys.dm_db_index_operational_stats(@dbid,@objid,@indid,@partitionid)

SELECT t.[text] AS [Adhoc Batch or Object Call],

SUBSTRING(t.[text], (qs.[statement_start_offset]/2) + 1,

((CASE qs.[statement_end_offset]

WHEN -1 THEN DATALENGTH(t.[text]) ELSE qs.[statement_end_offset] END

- qs.[statement_start_offset])/2) + 1) AS [Executed Statement]

, qs.[execution_count] AS [Counts]

, qs.[total_worker_time] AS [Total Worker Time], (qs.[total_worker_time] /

qs.[execution_count]) AS [Avg Worker Time]

, qs.[total_physical_reads] AS [Total Physical Reads],

(qs.[total_physical_reads] / qs.[execution_count]) AS [Avg Physical Reads]

, qs.[total_logical_writes] AS [Total Logical Writes],

(qs.[total_logical_writes] / qs.[execution_count]) AS [Avg Logical Writes]

, qs.[total_logical_reads] AS [Total Logical Reads],

(qs.[total_logical_reads] / qs.[execution_count]) AS [Avg Logical Reads]

, qs.[total_clr_time] AS [Total CLR Time], (qs.[total_clr_time] /

qs.[execution_count]) AS [Avg CLR Time]

, qs.[total_elapsed_time] AS [Total Elapsed Time], (qs.[total_elapsed_time]

/ qs.[execution_count]) AS [Avg Elapsed Time]

, qs.[last_execution_time] AS [Last Exec Time], qs.[creation_time] AS

FROM sys.dm_exec_query_stats AS qs

CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS t

ORDER BY [Counts] DESC