I was walking the exhibition floor today and had a profound experience. Looking over the shoulder of someone else, I was eavesdropping on a demo of a BI tool. It was a cool looking dashboard with impressive visualizations, and in the past few years, this would have been considered great technology, immediately applicable. However, what this customer asked next was worth the last couple years of hard work for me: "Will it take my colleague and I to build this or can users do it themselves?"

Wow.

Realize what has just happened. Our users now expect a lot more from the BI industry, to go beyond allowing customizations of pre-built solutions and instead enable more people to participate earlier in the food chain. This shift in the conversation is the best example of the inflection point we've now crossed. In my (biased) opinion, having a mainstream player not only enter the self service BI space but redefine what users expect from tools in this space is revolutionary. I don't use that word lightly: did anything else in the last decade move the dial as much to help realize the "BI for the masses" vision?

I hope you can appreciate the vigor and intensity we've been working with, and get some understanding for the focus. These are truly very exciting times!

It's been a while since I posted. I'm back in the Analysis Services team now where we've been busy finishing off SQL Server 2008 and working on a set of technologies that will be unveiled at the BI Conference.

I remember a conference back in the 7.0 days when we introduced OLAP Services and someone from the audience hurried outside to make a phone call. I overheard him saying "You need to see this... it's going to be big!"

This event will be one of those rare occasions where the team knows we're on to something that will change the BI market. If you're reading this and coming to the conference, besides a couple of keynotes, there is only one session that will talk about the technology in some detail. More info on that session is buried here. Until the conference, the abstract is the only information we can share, but believe me, we're itching to say more!  

If you do come by, I'll be in the back by the door waiting for people to make those calls. See you there!

ash

Chris, from Ordinal, sent an email earlier today announcing that they're done with a production ready version.

This is really great news, especially if you know the history of Ordinal and their reputation. Chris has been one of the most solid developers I've worked with either inside or outside Microsoft.

Anyways, check out https://www.ordinal.com/ssis.html if you're interested.

Enjoy,

ash

Over the last few months a lot of you have expressed a need to do impact analysis and lineage investigations across the different BI components we've got.... if I change this table here what packages, cubes, reports will be affected.

Well, earlier this week the BI team released a metadata pack that you can use for this. It includes sample projects, sample reports, etc. Right now it handles getting metadata from SSIS and SSAS, we hope to get to SSRS in the future. 

We decided to release the source code for it as well so that:

- you have a piece of code that demonstrates using the object models for SSIS and SSAS, and

- you can extend the work we've done for integrating other technologies in the metadata repository as well.

While the tool is useful as it is, we're hoping there'll be a community effort to drive this forward as well... I can easily imagine a workspace on gotdotnet where active participation will make this tool more comprehensive than we could ever manage to do.

Anyways, the link is https://www.microsoft.com/downloads/details.aspx?FamilyId=11DAA4D1-196D-4F2A-B18F-891579C364F4&displaylang=en

There's also a whitepaper here: https://www.microsoft.com/downloads/details.aspx?FamilyId=182BD330-0189-450C-A2FE-DF5C132D9DA9&displaylang=en

As usual, I look forward to your feedback.

enjoy

ash

For those not familiar with the excellent whitepaper on this, check this one out: https://msdn.microsoft.com/library/default.asp?url=/library/en-us/bts_2004wp/html/14bc36a8-69a9-48ed-8e4c-1c85202544c0.asp

regards

For those familiar with the changes in this release, my goodness. This has been a huge release. I've had a kid along the way, some have had two. Shipping software of this magnitude is probably one of the most gratifying experiences, and I had to go around shake the hand of everyone that I met today and congratulate them. 5 years is a long time.

If you're one of our regular contributors and haven't received a thanks from me directly, apologies. Wish I could shake your hand.

Now, on to the cigar.

Grant from the Project REAL team found a great nugget:

Openrowset now allows the BULK clause to be specified which accepts a filename and format specification. An OLE-DB driver for flat-files or XML would also do the job but this is a nice work-around for the native stack. The only difficulty is getting your head around the format for the .fmt file but after that it 'just works':

    select *
    from openrowset(bulk N'c:\temp\test.txt', formatfile = 'c:\temp\test.fmt') as x

thanks, Grant!

"Using the following data I’m trying to get each node Element populated into their own variables.
XML:
<DataBases>
   <DataBase>
      <Name>Sales</Name>
      <Action>DBCC</Action>
      <FLOP>1</FLOP>
   </DataBase>
   <DataBase>
      <Name>Inventory</Name>
      <Action>DBCC</Action>
      <FLOP>1</FLOP>
   </DataBase>
</DataBases>

Want to do:
Name   -> vsDataBase    
Action -> vsAction      
FLOP   -> vnFLOP
                      
Where vsDataBase, vsAction and vnFlop are variables of type string, string, int32. "

He was using For each loop's ForEach NodeList enumerator, lets see how we can do this.

There might be other ways to do it, I wanted to do it using two steps... first get a database, then gets its element values.

The result of the first step would be the nodelist, so the EnumerationType should be ElementCollection. Then there were the Outer* and Inner* properties that I confess I didn't understand.

Here's how this enumerator currently works:

- The OuterXPathString will be used to get the list of elements collection from the XML document specified earlier. In this case, our XPath string is '/DataBases/*' which would return all DataBase elements.

- For each element in the collection we now have (i.e. for each DataBase element), the InnerXPath is applied. We can set this property to be '*' (or the equivalent 'child::*') and InnerElementType of 'NodeText' so that we get the value of the child elements. Since there're multiple children per DataBase, the result of the this XPath will be an array of strings (each string is the value of the element).

For the first time through the loop, then, the current value of the enumerator is an array of three strings: {"Sales", "DBCC", "1"}. The next time through the loop the current value will be an array of three strings again: {"Inventory", "DBCC", "1"}.

Per the discussion above the current value of the enumerator during an enumeration should be an array of strings, but I wanted to see the array to make sure that was right. Turns out I had to use an undocumented feature to do this. (As soon as I post this message I'll be sending a note to the doc folks to make sure this isn't undocumented for too long because it's really useful!).

If you map the index '-1' to a variable of type 'Object', that variable gets populated with the complete collection (which is current value of the enumerator). Then I used a script task in the loop with simple code like so:

        Imports System.Collections
. . .

        Dim o As IEnumerator
        Dim o2 As Object
        o = CType(Dts.Variables("Variable").Value, IEnumerator)
        o.Reset()
        o.MoveNext()
        o2 = o.Current
 
and put a breakpoint at the last line.

When you execute the package now, the breakpoint is hit and the first time through you can see that o2 is the string "Sales". Just step through the iterator to confirm the array contents are what you expect them to be.

Knowing the contents of the array, then, it's fairly straight forward to do the mapping. Create the variables, set their types to string, then use the indices like this:

User::vsDataBase    0
User::vsAction      1
User::vsFLOP        2

Note that vsFLOP is not vnFLOP... the text is string, so you'll have to convert it to a number.

Also note that this example assumes a well formed document. Specifically that each DataBase element will have three children in the right order.

Think that's about it.

Is there a better way to do this in a loop? Let me know!

Thanks Fred for a great question.

ash

The desire that some folks have is to support parameters in their SQL statements. Sounds reasonable. The OLEDB source, for example, supports parameters. However, some providers don't support derivation of parameters from a statement. The question is how to make parameterized queries work for those providers.

For the OLEDB Source, the trick is to use the 'SQL Command from variable' access mode. What that means is that the sql command used during runtime will come from the contents of a variable. As you may know, in addition to just storing values, Variables can also store expressions which are evaluated to return the variable's value.

Lets work through an example. Lets say you want to execute this statement:

      SELECT Income from Customer where Age > ?

But your provider doesn't support finding out there's one parameter in this statement.

What we can do here is to create two variables. Below we have the properties you'd set for the two variables displayed in a '<prop>:<value>' format.

#1:

   Name: AgeVariable

   Type: String

   Value: 10

#2:

   Name: SqlStatementVariable

   Type: String

   Expression: "SELECT Income from Customer where Age > " + @AgeVariable

   EvaluateAsExpression: True

In this case I've hard coded the variable value to 10, but of course you can use whatever mechanism you want to (result of an Execute SQL task, the current value of a loop enumerator, etc.) to set that variable value.

Now we can use the 'SqlStatementVariable' as the source for the OLEDB Source.

How about on the destination side? What if I'm inserting into the customer table and want to set Age to 10, lets say, for all the rows.

The trick there is to introduce the variables as columns inside your data flow using something like a Derived Column transform and then map this column to the colunms of your table.

Hope this helps. If not, let me know and I can edit this post to make it more useful.

regards,

ash

First off, Performance:  

1. [Oledb Source, Lookup, Fuzzy Lookup]: Remove columns which’re not used downstream. 
2. [Oledb Source, Lookup, Fuzzy Lookup]: Use a select statement when fetching data from a view instead of using the table drop down.
3. [OLEDB Command Destination]: for large number of rows, this component might not scale because it sends one SQL statement per row. Persist data to a temporary table and use set based SQL.
4. [SCD]: for large number of rows that will not exist in the dimension table, consider using a lookup before the SCD.
5. [OLE DB Destination]: if the server is local, consider using SQL Server destination.

6. [Flat file source]: turn on fast parse for columns that have types that fast parse understands. Look at BOL for more info on Fast Parse. 
7. [Conditional Split]: For transforms that use conditions based on columns coming straight from OLEDB or ADO.Net sources, consider using the native filtering from the relational database to remove rows before they come in to the pipeline.
8. [Flat file Source]: For transforms that have all columns with the default column type and size, consider using the ‘Suggest Type’ functionality.
9. [OLEDB Destination]: Review the fast load settings on the destination adapter.
10. [SQL Server]: for OLEDB Destinations/SQL Server Destinations that perform bulk insert into a database, verify that the logging mode is appropriate for performance. Refer to BOL for more info.