The next generation of Big Data will focus on yet another set of challenges. Challenges of near real-time analytics, data quality and packaged analytical solutions (both predictive and descriptive) for business processes like CRM, Sales and Distribution, Marketing, etc.

1. Speed

Data is growing at an exponential rate, and the ability to analyze it faster is more important than ever.  Microsoft StreamInsight has been there for quite some time now.
Almost every big data vendor is coming out with product offerings, like in-memory processing to process data faster. Hadoop also launched its new release, Hadoop 2.0 / YARN, which can process data in near real-time. Another big data technology gaining traction is Apache Spark, which can run 100 times faster than Hadoop.

2. Data Quality

Data quality has never been as important as it is now with data growing at an exponential rate. The speed at which decisions are made has already reached a point where the human brain can’t keep up. This means that based on defined rules, data is cleansed and processed and decisions are made, all without any human intervention. In such environments, a single stream of bad data can act as a virus and result in incorrect decisions or heavy financial loss. A good example is the world of algorithmic trading, where trades are placed every few milliseconds by analyzing stock market trends using algorithms versus a human.

Data quality has become a key part of service level agreements (SLAs) in evolving digital enterprises. Bad quality data can result in blacklisting the data provider/supplier or severe financial penalties. B2B environments are the early adopters as they rely heavily on the quality of data to ensure smooth business operations. Some enterprises are moving in the direction of deploying real-time alerts for data quality issues. The alerts can be sent to the designated person based on the issue and can also suggest recommendations on how to fix the issue.

Machine learning is another technique that is being used to improve data quality. It has made it easier to conduct pattern analysis to identify new data quality issues. Machine learning systems can be deployed in a closed loop environment where the data quality rules are refined as new quality issues are identified via pattern analysis and other techniques.

3. Big Data Applications

Big data has created so much excitement that everyone wants to use it, but the technical challenges prevent greater adoption. Applications help overcome this challenge by making it easy for everyone to benefit from big data. Over the next few years we will see thousands of specialized applications launched for various industry verticals to solve big data challenges. In the near future, even a small business will be able to benefit from analyzing big data without requiring special infrastructure or hiring data scientists.

These applications will correlate customer data from multiple channels to have a better understanding of customers and hence will make more money for businesses by targeting the right customers with the right products. And, hopefully, some of these applications will make our lives better by having personalized applications for health care, diet / food, entertainment, etc.

In this post, I am going to describe how easily a hive query can be executed against a compressed data file. All screen shots are from my HDInsight cluster.

I am also using the new Windows Azure Powershell tool to interact with my HDInsight cluster. This is pretty straight forward and simple to interact.

(1) I first used the following commands to connect to my HDInsight cluster

Although I have used "Add-AzureAccount" to authenticate myself on the interactive console. The same can also be scripted using certificates.

(2) Next, I created a sample table by invoking Hive for the PowerShell

Notice that I have created an external table in hive as I am not doing any aggregation in this example.

(3) Now I will move compressed file in this location so that I can run hive query on this data. I am using Cloud Explorer to move file. Cloud Explorer also has a free version and presents a windows explorer like interface to interact with azure blob storage.

SampleZipFile.zip contains a bunch of CSV files all compressed together. The files are copied to the same location where the external hive table is point to.

(4) Now that data is in there, I can simple query this data using hive query. Hive will automatically run map/reduce jobs on the data nodes which hold a chunk of this compressed file!!

That's it !!!

Why should you care?

Recent Gartner reports suggest that 85% of the modern data is generated in new data types – those that were traditionally not stored in data warehouses. By 2015, organizations integrating high-value, diverse, new information types and sources into a coherent information management infrastructure will outperform their industry peers financially by more than 20%. Organizations today are ramping up their IT infrastructure and skills to take on this challenge. Big Data COE are getting formed to design, architect and implement such solutions.

Just to illustrate why it makes sense, it will be exciting to know how organizations today can analyze huge pile of emails sitting in their exchange servers to find out how teams collaborate with each other, if at all they are!! You would agree that most projects in an organizations fail due to improper or lack of communication. And there are numerous examples – sentiment analysis for marketing purposes, political campaigns, etc.; click-stream analysis for digital marketing, customer behavior, personalized advertising, targeted surveys, etc Opportunities are endless for organizations to innovate.

Following example shows how IT communities are connected in different parts of the country. Communities which are closely connected have better collaboration. This is some manner influences how skilled the resources are. Most recruitment companies today are doing such analysis.

 The data shown below is extracted from a social networking site called GitHub (a social network site for open code).  The network graph shows that Bangalore is much better connected than Delhi/NCR or Pune.  

Also, Bangalore and Singapore are somewhat similar and have pretty closely knit IT community.

How to you build these?

HDInsight is Microsoft’s implementation of Hadoop on cloud. A 100% apache Hadoop service that you can spin-off in just a few minutes. HDInsight offers enterprise-class security, scalability and manageability. Thanks to a dedicated Secure Node, HDInsight helps you secure your Hadoop cluster. You can also take full advantage of the elastic scalability of Windows Azure. In addition, HDInsight simplifies manageability of your Hadoop cluster through extensive support for PowerShell scripting.

Analysis is seldom done in isolation. It is usually, and almost always, done in conjunction with structured data. How do you combine the two together without transporting neither the results nor the raw data? Microsoft Power Query and PolyBase are innovative and next generation BI tools that allow organizations mesh-up semi-structured, unstructured and structured datasets into a single unified data model. Power Query can directly connect to Hadoop distributed file system (HDFS) running on=premise or on Azure (HDInsight). PolyBase in its v1 avatar, can connect to HDFS running on premise. This truly redefines Business Analytics in the modern data warehouse.

First and foremost, let us understand how memory is used by SSAS.

• Processing: During processing, SSAS needs memory to load data and create dictionaries and related data structures before it flushes them to disk. In addition, if the database being processed already contains some data, it must hold the previous version of the database until the transaction commits and the new database is ready for query.
• Querying: During a query, SSAS sometimes need memory to hold temporary data structures that are needed to resolve the query.

SSAS configuration file msndsrv.ini stores memory settings for Tabular model. The easiest way to view and change these setting is by opening Analysis Services properties page inside SSMS.

Following screen shows the default values of these memory settings.

VertipaqPagingPolicy: This is a Boolean property with 0 and 1 as possible values. In mode 0, all xVelocity data is locked into memory, whereas in mode 1, xVelocity in-memory engine pages data to disk if the system is running out of memory. More specifically, in mode 1, only hash dictionaries are locked in memory; data pages can be flushed to disk which enables xVelocity to use more memory than is available. Must keep in mind that paging has detrimental effect on performance. Default value of this property is, mode 1.

HardMemoryLimit: This is the maximum memory that SSAS can ever allocate. If SSAS exceeds the hard memory limit, the system aggressively kills active sessions. With VertipaqPagingPolicy in mode 0, it is also the limit for the maximum working set of the process. If HardMemoryLimit is set to 0, it will use a default value mid-way between HighMemoryLimit and total physical memory.

TotalMemoryLimit: If memory usage exceeds the TotalMemoryLimit, memory manager evicts all cached data that is not currently in use. TotalMemoryLimit should always be less than HardMemoryLimit. The default value 80% of physical memory or the virtual address space, whichever is less.

LowMemoryLimit:  This is the point at which the system starts to clear caches out of memory. As memory usage increases above the LowMemoryLimit, SSAS becomes more aggressive about cycling cached data until it hits the total memory limit, at which point it evicts everything that is not pinned.

VertipaqMemoryLimit: If you choose mode 0, the VertipaqMemoryLimit defines the total amount of memory xVelocity is allowed to lock in the working set. In mode 1, it defines a limit for the physical memory that is used by xVelocity, allowing paging for the remaining memory above this limit.

If you want to reduce memory for an instance of SSAS, it makes sense to set VertipaqMemoryLimit to a value that is lower than LowMemoryLimit.

No memory cleaning happens below the LowMemoryLimit, and the aggressiveness increases as soon as the usage approaches TotalMemoryLimit. Above TotalMemoryLimit, SSAS gets committed to cleaning memory, even if panic mode starts only after HardMemoryLimit.

All limit values are expressed as numbers. If their value is less than 100, it is interpreted as percentage of the total server memory. If it has value greater than 100, it is interpreted as the number of bytes to allocate.

Let us use an example to illustrate this. Say memory limits for a particular SSAS tabular instance are configured as under:

(a)    TotalMemoryLimit = 120 GB

(b)    LowMemoryLimit = 110 GB

(c)    VertipaqMemoryLimit = 100 GB

At some stage, assume that xVelocity data structures are using 210 GB and the process‘s total memory usage is 215 GB. These usage counter are way above the TotalMemoryLimit, however, because VertipaPagingPolicy is set to 1, memory eviction in this case will not happen.

This is because,

Total xVelocity memory pinned =  VertipaqMemoryLimit = 100 GB. Remaining 110 Gb will be paged to disk. Memory used by other SSAS processes = 215 GB – 210 Gb = 5 GB. Therefore, total memory used by SSAS which is pinned = 100 GB + 5 GB = 105 GB. Since this is below LowMemoryLimit, no memory eviction will happen.

Data Explorer is an Excel plug-in that helps users discover data and convert it into a shape or form that they need. Data Explorer is only available as a preview at this point and works with Excel 2010 SP1 or Excel 2013.  You can download the Data Explorer preview from http://www.microsoft.com/en-us/download/details.aspx?id=36803 The plug-in is available for both 32-bit and 64-bit version of Excel.

Why do we need it?

Data Explorer provides a simple and easy to use interface for data analysts and data scientist to:

Explore data from any source – Data Explore provides connectivity to a wide variety of data sources such as databases, file, SharePoint Lists, Web, Facebook, OData Feed, Windows Azure Marketplace, SQL Azure database, Hadoop HDFS and many more.

Convert data in the shape and form required for analysis – by performing basic ETL and aggregation operations themselves using a simple interface. Operations such as group by, split, join, union, calculated column, lookups, data type conversion and many more.

Combine data from multiple sources – to create a data model for analysis

Improves self-service BI – by loading Data Explorer model to BI tools like PowerPivot and Power View.

Data Explorer will greatly reduce the time it takes for data to get prepared before it Is loaded into data warehouse and becomes ready for analysis.

Lets Get Started !!

A complete documentation of Data Explorer his published ere http://office.microsoft.com/en-us/excel-help/start-page-HA104003813.aspx

In my example here, I will take SQL Saturday event list as my base data that is published on the SQL Saturday web site http://www.sqlsaturday.com/events.aspx

Let us break our data preparation in following smaller tasks

(1)    Get the list of events from SQL Saturday web site

(2)    Convert this data into the form we need for analysis

(3)    Use Windows Azure Marketplace (free) services for lookup of additional data

(4)    Load this query in data model

(5)    Analyse this model using Power View

Task 1: Get the list of events from SQL Saturday web site

 1)      Open Excel and select Data Explorer in the top ribbon. This will open up the Data Explorer menu bar.

2)      Select “From Web” under Get External Data

3)      Enter URL for getting SQL Saturday event list, http://www.sqlsaturday.com/events.aspx

4)      Click Apply and you will now get a list of data tables in the left pane. Both Table0 and Table1 essentially have same data with Table0 having only one additional “Null” column. We will select Table1 for this example.

5)      Click on Table1 and you will now be able to see the event data. Exploring data from a web site in excel has never been so simple before!!

6)      Click “Done” and the data will be loaded in a new worksheet.

Task 2: Convert this data into the form we need for analysis

1)      Click “Filter & Shape” to open Edit Query window once again. We will now perform some basic transformation operations

2)      Hide “Column1” by doing right-click and selecting “Hide” option.

3)      Now we will split “Column3” to extract event ID and event name.

4)      Right click “Column3” and select “Split Column by Delimiter “.

5)      Under “Select or Enter Delimiter” select custom and enter delimiter character as “#”.

6)      On clicking “Apply”, column 3 will be split into column 3.1 and 3.2

7)      Follow steps (4), (5) and (6) to split column 3.2 to get event ID and event name using “-“ as delimiter.

You can also write Data Explorer formula for doing this operation,

= Table.SplitColumn(SplitColumnDelimiter,"Column3.2",Splitter.SplitTextByDelimiter("-"),{"Column3.2.1", "Column3.2.2"})

Notice that column 3.2.1 and column 3.2.2 are created representing “Event ID” and “Event Name” respectively.

8)      Hide column 3.1 by doing right-click -> Hide or by entering following Data Explorer formula

= Table.RemoveColumns(SplitColumnDelimiter1,{"Column3.1"})

9)      Rename all columns by doing right-click on each column -> Rename, or double-clicking the column header, or using following formula

= Table.RenameColumns(HiddenColumns1,{{"Column2", "Event Date"},{"Column3.2.1", "Event ID"},{"Column3.2.2", "Event Name"},{"Column4", "Event Location"}})

10)   Convert column data type using right click -> Change Type -> Date (or target data type) as shown below.

Or using following formula

= Table.TransformColumns(RenamedColumns,{{"Event Date", Date.FromText},{"Event ID", Number.FromText}})

11)   Click “Done” and you have your shaped-up data in excel now.

Task 3: Use Windows Azure Marketplace (free) services for lookup of additional data

 So far we have been able to get SQL Saturday data in excel and shaped it by extracting event IDs and event names. A look at, “Event Location” column reveals that the location data is pretty much a free text. We need to extract city, state and country information out of this free text in order to do some meaningful analysis.

I am going to use the free data quality service available on Windows Azure Marketplace to achieve this task. This service gives me pretty good results.

1)      Subscribe to Windows Azure Marketplace “Melissa DataAddress Check” service using your Microsoft account.

2)      Click “Filter & Shape” to open Edit Query window once again. We will now a new query step called “AddressLookup” for dynamically invoking <alesis Address Check service for “Event Location”

3)      Click on formula bar and enter following formula for invoking this service as OData Feed.

= OData.Feed("https://api.datamarket.azure.com/MelissaData/AddressCheck/v1/"){0}[Data]

4)      Click on “Edit Credentials” to enter credentials for accessing this service. I am using fast key for this demo.

5)      If authentication is successful, you will receive an “Invoke” page as shown below.

6)      Rename the above step to “AddressLookup” byright-clicking “Custom1” in right pane

7)      We will now invoke this step dynamically for each row and place the results in a new column, called “Address”

8)      Click on formula bar and enter the following Lookup formula

= Table.AddColumn(RenamedColumns,"Address",each AddressLookup([Event Location],1,0))

You are likely to receive a warning as shown below.

The reason for this warning is that Data Explorer tries to optimize performance of this look-up using xVelocity, and it finds this feature “Fast Combine” disabled by default.

9)      Click “Done” to close query editor.

10)   Select “Fast Combine” menu option inside “Options” group.

11)   Click “Enable”

12)   And open query editor again. You will now see your new column “Address” populated in the query window. For each row in the list, Windows Azure Address service is invoked and the results are received using OData Feeds!! All of this is done using a simple formula and the results are available with you in excel. This is awesome!!

13)   But you would notice that instead of seeing actual data, you see “Table” in Address column. What is that?This simply means that a temporary table has been created containing the data received from OData Feed.

14)   Click on the small icon next to column header

15)   This will show you a list of columns. You can choose which columns you want to view in your model from this window.

16)   Data Explorer will now import the selected columns in the data model. After you click “Done”, you will be able to see this data in your excel worksheet.

In a couple of clicks, we have extracted zip codes, city, state, country, latitude, longitude and other details from a free text using excel!! Isn’t this cool!!

Task 4: Load this Query in Data Model

You are now ready to load this query in data model (data-warehouse) and make final settings to make this ready for analysis.

1)      On the worksheet, click on “Show Query Settings” so that you can see the query setting pane on the right

2)      Click “Load to data Model” hyperlink

3)      The query is now loaded as a PowerPivot data model.

4)      Select PowerPivot from top menu bar and click “Manage” in data model group as shown below

5)      You will find your model appearing in the PowerPivot wndow.

6)      Select “Address.Confidence”, “Address.Latitude”, “Address.Longitude” and “Address.ZipCode” columns and set data type to “Decimal Number”.

7)      Select Address.Latitude and set data category to “Latitude”. Repeat the same for Address.Longitude

8)      Add filter criteria to Address.Confidence. Select only those rows which have confidence value greater than 0. The filtered list will lokk like this

9)      The model is now ready to be analysed. Select PivotTable and create a new PivotTable form PowerPivot window

Task 5: Analyse Model using Power View

We can now view this data in Power View for Excel and create rich visualizations and analyse this data.

1)      Click Insert from the top menu and select Power View.

2)      Select following data points from the model:

1. a.      Address.Latitude
2. b.      Address.Longitude
3. c.      Event Date
4. d.      Event Name
5. e.      Event ID

Change view type from table to Map. You will now see this data represented on the map of US.

Microsoft BI platform has a comprehensive set of data exploration and reporting tools that truly democratizes the concept of business intelligence in an organization. While these gives you insights at different level, SharePoint allows getting these together and create a mesh-up of these incredible visualizations using web parts.

Web Parts are server-side controls that run inside the context of site pages. SharePoint provides a number of web parts out-of-the-box and also allows developers to create custom web parts. Web parts are editable and configurable by users.

Power View reports can be integrated as web parts on a SharePoint site page using of the two methods:

Using Page Viewer Web Part

Page Viewer is a generic web part that you can use to display a Web page using a hyperlink. This method is pretty straight forward and you can follow these steps to enable it:

1)      Create a new Page or open an existing one in edit mode

2)      Select “Insert” from the top menu and click “Web Part” in parts group

3)      You will now see a list of categories on left side and associated web parts on the right of it.

4)      Select “Media and Content” under category and “Page Viewer” under parts

5)      Click “Add”

 6)      Page Viewer web part will now be added to the page.

7)      Click “Open the tool pane” to start editing the web part

8)      In the URL text box, paste the Power View report URL. The one I am using for this example looks like this,

http://sql2012sp1/_layouts/15/ReportServer/AdHocReportDesigner.aspx?RelativeReportUrl=%2fPowerPivot%2520Gallery%2fAuto%2520Sales%2520Sample%2520Report.rdlx&ViewMode=Presentation&ReportSection=ReportSection&True&Fit=True&PreviewBar=False

Notice the values and options used in query string:

• ·    RelativeReportUrl: Refers to the relative path of the power view report. For example, %2fPowerPivot%2520Gallery%2fAuto%2520Sales%2520Sample%2520Report.rdlx
• ·    ViewMode: Can be set to either Presentation or Edit
• ·    ReportSection: Refers to a view within the RDLX. Each view in your Power View report has a reportsection identifier. The default view is named as “ReportSection” and any new view that you add is named as “ReportSection2” and so on. This option only determines which view to show as default. Users will still have options to navigate to other views by using the navigation controls.
• ·    Fit: Allows Power View report to automatically resize itself to fit within the web part frame
• ·    PreviewBar: Allows you to show or hide the preview toolbar on top. While presenting Power View report on a dashboard page, it is a good idea to hide toolbar, unless you want users to edit the report as well.

9)      Set other web part parameters such as height, width, layout, etc and

10)   Click “Ok” and you will now be able to see your Power View report as a web part. A sample SharePoint web page with Power View report as a web part is shown below.

Limitations – Page Viewer web part can only be used in a browser that supports the HTML <IFRAME> element..

Using Silverlight Web Part

In this approach, we will use a generic Silverlight web part to host our Power View report. This web part allows users to deploy Silverlight application on SharePoint without the need to create a custom web part for every new application. The Silverlight web part provides an option for user to specify a comma separated string to specify the initialization parameters.

(1)    Create a new Page or open an existing one in edit mode

(2)    Select “Insert” from the top menu and click “Web Part” in parts group

(3)    You will now see a list of categories on left side and associated web parts on the right of it.

(4)    Select “Media and Content” under category and “Silverlight Web Part” under parts

(5)    Click “Add”

(6)    You will now be prompted to enter Silverlight application package. If your power view content type is enables and Reporting Services service application is running, Silverlight application package can be invoked from the following URL

/_layouts/ReportServer/ClientBin/Microsoft.Reporting.AdHoc.Shell.Bootstrapper.xap

(7)    Enter this URL and click “Ok”

(8)    A Silverlight web part will now be added to your page.

(9)    Select “Edit Web Part Settings” from the web part drop down

(10) Edit pane will appear on the right side as shown below

(11) Set appearance and layout parameters

(12) Expand “Other Settings” to enter custom initialization parameters

 (13) Initialization parameters for Silverlight web part has to be provided in the following format

ItemPath=http://<YourServername>/<YourSite>/YourReport.rdlx,ReportServerUri=http://<YourServername>/_vti_bin/reportserver/,ViewMode=Presentation,PreviewBar=False,Fit=True

The one I am using for this set up is,

ItemPath=http://sql2012sp1/_layouts/15/ReportServer/AdHocReportDesigner.aspx?RelativeReportUrl=%2fPowerPivot%2520Gallery%2fAuto%2520Sales%2520Sample%2520Report.rdlx,ReportServerUri=http://sql2012sp1/,ViewMode=Presentation,PreviewBar=False,Fit=True

11)   Click “Ok” and you will now be able to see your Power View report as a web part. A sample SharePoint web page with Power View report as a web part is shown below.

What is Dynamic Security?

In my view, dynamic security is a technique that enable a single role to apply different permissions for different users based on a business condition. For example, while User A and User B are assigned a role called ReportReader, User A should only see data for North America whereas User B should see data only for Asia.

This type of security is a common requirement in any enterprise BI implementation.  It is a handy technique where you would otherwise be forced to create and maintain a large number of roles.

DAX Functions for Dynamic Security

DAX provides two very useful functions for implementing dynamic security:

• ·        USERNAME: returns name of windows user connected to the tabular model
• ·        CUSTOMDATA: returns string value passed to CustomData property of connection string

Let us see how these work using a sample code, I open SSMS and connect to tabular SSAS model. I am setting CustomData property to “Hello World” in the connection string.

In SSMS, I can now open MDX query editor and write this evaluate expression to read these values

EVALUATE

ROWS (

               “Windows User: “, USERNAME(),

               “Custom Data: “, CUSTOMDATA()

)

 Important thing to note here is that these functions can return different values for different users, which means that same row filter in a role can return different results for different users.

Implementing Dynamic Security using CUSTOMDATA

The CSTOMDATA function is useful for dynamic security when a front-end application handles authentication of users but must push the job of applying data security back to Analysis Services. For instance, an internet facing reporting tool might not want to rely on Windows authentication and instead have its own system for authenticating end users. It would then open a connection to Analysis Services (as a single Windows user) for each query that was run, but each time it opens a connection, it would pass a different value through the Customdata connection property to indicate the end user for whom the query was run. Analysis Services can then use this value as part of row filter expression.

CUSTOMDATA is not suitable for dynamic security when end user connects to tabular model directly because in that case, an end user might be able to change this property in whatever client tool he or she is using.

To demonstrate the use of CUSTOMDATA, I will use following code

(1)    First I will create a new role called CustomDataRole by selecting menu item Model -> Role. Click “New” to create a new role and name it CustomDataRole

(2)    Enter filter expression for Geography table

= IF (CUSTOMDATA() = “”, FALSE(), Geography[Country Region Name] = CUSTOMDATA())

(3)    Now open SSMS and set CustomData parameter to a value such as “Canada”

Open a new query window and test this out. You will now see data only for [Country Region Name] = “Canada”

Implementing Dynamic Security using USERNAME

The USERNAME function is used to implement dynamic security where end users connect to a tabular model directly, with Windows credentials. Because one user is likely to need access to many rows on a table, and one row on a table is likely to be accessible by more than one user, a permission table approach is usually necessary when this flavour of dynamic security is used.

To illustrate this,

(1)    I will use a UserPermission dimension

(2)    Next I will create a new role called UserNameDataRole with “Read” permission and add above three users to it

(3)    Now I will add row filter expression for Geography dimension as follows

= CONTAINS (

               UserPermissions, UserPermissions[userid], USERNAME(),

                              UserPermissions[country], Geography[Country Region Name]

               )

(4)    Open SSMS and set following parameters

Roles= UserNameDataRole; EffectiveUserName=contoso\Adam

(5)    Now run DAX query,

EVALUATE Geography

You will see that only rows for “Country Region Name” as “United Sates” will appear.

We can now check the same for contoso\Scot

 In the next blog post I will discuss kerberos and double-hop scenarios.

Defining a separate date table is a common practice in any star schema and this is a suggested technique for any tabular model. Thus, whenever you have a date column that you want to analyse, you should create a relationship with a date table. If you have multiple date columns in a table, you can create multiple inactive relationships to the Date table plus a single active one.

Creating a Date table

An obvious question will be that what is the best way to create a Date table? Well, you have atleast three options:

• ·        Generate Dates in a SQL Query or View
• ·        Generate Date table in MS Excel and import it in your tabular model
• ·        Import a Date table from DateStream on Windows Azure Marketplace

Generate Dates in a SQL Query or View

While there are many ways to create this, I normally use this script

Good thing about this script is that, it has Date_ID column which is int, making it possible to relate it to date columns in any format.

Generate Date table in MS Excel and import it in your tabular model

You can create a column containing dates and then add other columns like day, month, year, etc using Excel formulas that produce these values.

Import a Date table from DateStream on Windows Azure Marketplace

Datastream is a free data source available on Windows Azure marketplace whose source code and documentation is available on http://datastream.codeplex.com

Setting Metadata for Date table

Many DAX functions that operate over dates requires a Date table having a Date Type column. In order to mark a table as Date table,

(1)    Open your tabular model in SQL Server Data Tools

(2)    Select your Date table

(3)    Right-Click and select “Mark as date Table”

(4)    Select the Date Type column and click “Ok”

Year-to-Date, Quarter-to-Date and Month-to-Date functions

These calculations are pretty common during analysis and are very similar in nature. While Month-to-Date function is used for analysing data at the day level, quarter-to-date and year-to-date functions are used for analysing data at the month level.

One way to achieve this, is to use CALCULATE and DATESYTD functions.

[YTD Sales] := CALCULATE([Internet Total Sales], DATESYTD(‘Date’[Date])

DATESYTD is a built-in time intelligence function that returns a list of all dates from the beginning of the year until the last date included in the current filter context. This list is passed as an argument to the CALCULATE function to set the filter for the Internet Total Sales calculation.

Since this patter is very common, DAX offers a dedicated function that simplifies this syntax!!

[YTD Sales] := TOTALYTD ([Internet Total Sales], ‘Date’[Date])

[QTD Sales] := TOTALQTD ([Internet Total Sales], ‘Date’[Date])

[MTD Sales] := TOTALMTD ([Internet Total Sales], ‘Date’[Date])

Period from Prior Year

Users and analysts usually need to get values from a period of the prior year. This is useful for making comparisons of trends during a period last year to the same period this year. Thus is the DAX function SAMEPERIODLASTYEAR that returns a set of dates shifted one year back in time. This is a specialized version of the more generic DATEADD function.

[Last Year Sales] := CALCULATE ([Internet Total Sales], SAMEPERIODLASTYEAR (‘Date’[date]))

Sometime you may want to look at a measure for the previous year, usually to compare it with the year-to-date total. To do that, you can use another specialized function:

[Last Year Sales] := CALCULATE ([Internet Total Sales], PARALLELPERIOD (‘Date’[date], -1, Year))

It is a user-driven, computer-assisted solution to identify incorrect or incomplete data and modify, remove, or enrich that data to resolve data quality issues.

You can integrate DQS as a component of SQL Server Integration Services (SSIS) and into Master Data Services (MDS) work-flows. The SSIS data cleansing transformation applies rules that have already been configured in DQS to a data flow. This allows users to send data from a SSIS work-flow to cleanse before sending to the target destination.

Data Quality Services.docx

• ·        Rapid provisioning
• ·        Cost-effective scalability and High availability
• ·        Reduced management overhead
• ·        Secure access for viewing and managing reports.

Windows Azure Reporting offers the following capabilities:

• ·        Host operational reports based on report data from existing Windows Azure SQL Database. Operational reports are report definition language (RDL) reports.
• ·        Embed the Visual Studio Report Viewer AJAX control in on-premise Web applications or cloud-based Windows Azure applications to add rich reporting to custom applications. The Report Viewer control points to reports hosted on report servers.

The document provides a step-by-step guide on how you can migrate your existing SSRS reports and ASP.Net application to Windows Azure.

Building Windows Azure Reporting Application.docx