Like all Microsoft employees I had access to a survey about the name a few months ago. I don’t remember anymore what kinds of choices I had, I remember not being very excited about them. But I do remember that it was clear we were going to be positioning PowerPivot as an essential part of the IW (information worker) experience. The marketing materials certainly back that up. But most importantly, from my experimentation with the CTPs and the internal news I hear, I think the product backs that up as well.

I think that’s why it gets branded with a name that references PivotTables, because one of the primary goals is making IWs able to build data models more easily. I originally thought the name options along those lines were too limiting, but I think it’s more and more clear how well SQL and Excel are working together to ensure there is powerful modeling and analysis capabilities available for everyone (in a SQL + SharePoint + Office environment). A name that gave more credence to the engine would not be relevant at all to the IW, so this name communicates our goals better.

Then so you don’t think the SQL team is the only team driving BI for IWs, you might be interested in some more details about how new features in Excel 2010 PivotTables. With all the emphasis on PowerPivot naturally landing in Excel and reusing a familiar environment, the PivotTable team has to keep up appearances as well! Most of these improvements are relevant to OLAP-ish scenarios, but especially the calculation options are meant to let ordinary people (IWs) do analysis, and not just OLAP experts.

For performance of the AddIn not too much changed, although on scenarios involving large submissions (10s of thousands of cells in the change list at a time) there were some improvements to reduce the size of the submission which yields a faster submission. Also the memory consumption is reduced during this step so fewer contributors should have problems making large submissions. On moderate and small sized submissions I think there is no change, although its possible for tiny submissions the extra process of shrinking the change list may add a little bit of overhead, but should not be noticeable.

For outside the AddIn I think the biggest changes are related to how security can be configured with PPSCmd, and the calendar maximum limit has been increased. The Monitoring and Analytics SP3 was also released, and if you missed it ProClarity’s latest SP was also released….

Asymmetric CrossJoins

I think PPS Planning’s Excel AddIn handled these in a very easy to understand fashion. Now Excel PivotTables has support for this as describe in this post. But Excel takes it one step further and allows for better reuse of the sets. Over all I think this UX of remaining consistent with earlier releases, but still quickly allowing people to delete the intersections that aren’t meaningful to them is pretty powerful.

Custom MDX

Indeed if you read that post fully, you’ll realize that Excel’s PivotTables couldn’t previously handle this at all because of their default aggregation features. To enable this PivotTables have to be a lot smarter about OLAP vs. other data sources. So now rendering arbitrary MDX is almost a hidden feature. But besides having a friendly UX to edit sets, you can also supply your own expressions. Really they have achieved one of the most highly desired features by MDX geeks. And one of the few reasons why I still sometimes have to answer questions about using the Planning Matrix instead of PivotTables for xl2007 users.

Dynamic Sets

This was one of the frequent asks by report authors. I remember working lots to enable truly dynamic rendering with respect to the filter in the matrix. Because Excel’s implementation of the above features is so complete, this one probably felt like a freebie for people reading about new features. But I know improving the rendering was no simple feat, but this is a big win for OLAP PivotTable users.

Slicers

Ok I can’t claim my team did everything before Excel, and don’t mean to. Although we had global filters, they are nothing compared to slicers. Follow that post and see how easy to use slicers are for the average report user. It’s just amazing! The layout is customizable and that themes are enabled. I think the Excel team did a fantastic job here, and report authors who currently are doing a lot of custom VBA coding and using form controls, really will save a lot of time in the future with slicers.

Business rules and Design-time formulas

I still think rules was an area where PPS Planning shined. I’ll grant you that the V1 UX for maintaining rules was quite a bit of work. But I know some people still found them too complicated. So they asked for design-time formulas in the AddIn. I think clearly my team’s AddIn didn’t get this one right. We missed realizing that what people really wanted was a way to express rules and calculations in a language they already knew: Excel formulas. The Gemini team I think is doing a great job here. The collaborated a lot with the Excel team and I think DAX looks great. Naturally there are a few differences, but check out this link to Data Analysis Expressions (DAX) and I think you’ll agree with me that this looks very cool.

If you take all these features (and of course PivotTable writeback) you have probably some 90% of the PPS Planning AddIn, as well as some of the features of Business Modeler. It’s not at all the same kind of product that PPS Planning was, but I think is clearly a platform that enables building the same kind of solution. (Remember, except for Gemini which also suffers a bit from the AddIn model, that everything implemented as a native feature of Excel is accessible through the object model!) It’s been so hard to sit quiet while news of Office 2010 was embargoed. But these features I call out here, are what made continue to have faith in my employer. (And I’m just cherry picking the features from Excel 2010 and Gemini that overlapped with PPS Planning scenarios.)

Anyway, I think these fears of Gemini are unwarranted because regular folks don’t have access to raw database tables. (Right? If they do, then I have a lot of fear about your systems.) The integration shown in demos with data feeds and Reporting Services reports looks much more like the kind of data sources that regular folks will consume. IT publishes data in a relatively bland report, which is fairly cheap and easy for them. Power users and some adventurous non-power users will consume these kinds of published models, not building them from scratch. (Ok some people will build them from scratch, but only if you give them access to raw tables. Which not a fault of Gemini. And power users are already beyond your control). In this scenario, yes DBAs who are building trivial models, need a little fear. But just enough to motivate them to move on to the harder problems.

Then we are back to the all-important single version of the truth. That holy grail that IT and DBAs seem to think has been achieved. There is great fear that Gemini will destroy that palace and the empire of IT will be overrun by trolls and gremlins. Ha! What I think the fear-mongers are missing is: end users are already building multiple versions of the truth that are different from the IT version of the truth! Even in places that have the strongest data warehousing implementations. Worst of all, this alternate-reality-version-of-the-truth data goes directly to senior management!

They are doing this with such dastardly tools as Excel and PowerPoint. But it’s so hard to get data into Excel, how can they possibly do this?! They are copy/pasting data or manually entering it while reading from another source. Then they build presentations and give them to senior management. Who is there to make sure that the data still matches the one-truth? Nobody. I just don’t believe that you present by to the board of directors by clicking refresh in a web browser to look at a dashboard. (Well, I wish you would since I work on PerformancePoint Services for SharePoint. Maybe, if you’re lucky, the audience goes back to their desk and does this, but I doubt it.) Anyway, my point should be clear: When people are truly making monumental decisions based on data, it does not come directly from the one-truth-palace, it comes from false prophets!

So how do I think Gemini helps instead of maintaining the status quo? With the data size minimized, there’s no reason not to include the data model in the workbook. That way it can be examined and vetted – especially if you leave the connections back to the data sources. Now the new version of the truth is documented and it has a clear link to the original version of the truth! So if you continue in this vein, why not embed the entire workbook+Gemini into the presentation? Now people with access to the slides also have the data, so they can dive into the assumptions instead of heckling from the back row. I think this is a huge advance. I don’t know if it was intended to bring the one-truth to presentations, but I think a Gemini-style solution is absolutely critical to a one-version-of-the-truth solution. It acknowledges multiple truths, but provides a mechanism to explain how and why this truth is better.

I think that’s pretty powerful, and I’m disappointed that commentators haven’t come to the same conclusion I am. Or maybe I’m wrong? Flame away if you have comments…I can take it… Well at least I can delete them if it hurts too much!

(Again, I’m not on the Gemini team, and I’m not a spokesperson for Microsoft in any shape. I’m just talking through through what I see, and why I am thinking differently than others about what Gemini can do for end-users.)

From my perspective of implementing the PerformancePoint Planning Excel AddIn, I can say Gemini would have been a huge help. The compressed size of Gemini models is amazing compared to a local cube file. Not to mention the hassle of maintaining the cached copy outside the workbook and assorted issues of how to keep workbooks in sync with the data. Plus the query speeds really are fantastic. There’s just that little detail of what-if queries…

In the reporting and data analyst space, I think Gemini looks like a huge help. I see lots and lots of fear about how Gemini will be managed and the mess that ordinary users will make. (Disclaimer: I’m not on the Gemini team, I’m just talking about my interpretation of all the demo’s I’ve seen and the experimentation that I’ve done.) With what is in the CTP I would not be afraid of ordinary users. They probably won’t create complicated data models yet so not too much can go wrong. And for reporting to always have the report and the data in sync, seems like a huge help. For scenarios where the data needs to be refreshed at specific intervals or on demand the SharePoint integration I think hides most of the complexity from most of the users. I’ll come back to fear-mongering in another post.

For data modeling, I think the Gemini user experience is a huge leap forward as compared to BIDS. I think this will open up some data modeling capabilities to regular folks. But obviously it doesn’t do everything, but it feels like the kind of tool that end-users can work with. The kind of thing they were already doing manually in Excel with lots of temp tables and macros and manual intervention. For DBAs maybe it still looks limiting, but they are welcome to continue using BIDS. End users want to give up the fragile temp tables and macros.

From the PerformancePoint Planning experience, I know people want an easier way to define business rules. PEL was an interesting step, but it didn’t go far enough. It was easier than MDX, but not as easy as Excel formulas. Gemini takes it to the logical conclusion of as close to Excel formulas as possible. Now I have great faith that business users can achieve the result of defining business rules without a DBA intervening.

Clearly Gemini is not a magic cure-all. But it’s a huge step in the direction of making end-users able to self-serve their simple BI needs. When the data already exists but maybe in different silos. Or they need to provide some assumptions that can’t be edited in a LOB system. Self-service doesn’t have to mean you can prepare a 10 course meal. But you can pour the soda into a cup, and pull a salad together from the buffet line without having to bring in a chef from Le Cordon Blue too. Gemini is definitely the right direction, it will be interesting to see how close they get to the 10 course meal…

Again, I’m not on the Gemini team, and I’m not a spokesperson for Microsoft in any shape. I’m just talking through what I see, and why I am thinking differently than others about what Gemini can do for end-users. Flame away if you think I’m crazy, maybe I won’t delete the comments.

My team has been mainly working on the Monitoring and Analytics components in SharePoint lately. Not many details are out about SharePoint yet either, so I'll keep quiet on that front. Anyway, most of our work has been under the hood, rather than new feature-related so I won’t have much to say.

So onwards to Excel 2010. The videos I've seen are covering the new Slicer and SparkLine functionality so you should check that out. I like both of them a lot. We got some feedback about how Planning's filters rendered, and I think having end users see all the available choices with slicers, rather than having to navigate to a drop down or through a tree is going to be much more user friendly.

What I haven't seen in videos yet, but am very excited about, is PivotTable what-if and writeback for OLAP (well at least for Analysis Services). I've had to keep quiet about this for a long time. But I've finally seen a brief mention in the official blog and some people are actually using it!

The PPS Planning team had some great opportunities to provide feedback about this feature to make sure it would meet our (expected) needs, as well as pass along the feedback we had about data entry in the matrix. This is probably 40-50% of the functionality in the Planning add-in that is must-have from customers and partners I've talked to. (another 30-40% I would say is workflow but that’s a topic for another post). I think it turned out quite well.

I know a lot of Partners are frustrated with the PPS Planning situation, but I really think you should look at Excel 2010 for your next generation solutions. Excel is now able to participate fully out of the box -- custom code is not required on the client for some of the hardest parts. And it is fully supported through the object model so if you need to have customize, or need to provide additional validation etc, it's all possible. Having writeback as a native feature of Excel is fantastic, so many customizations that weren't supported by the Planning add-in are possible. Plus you get full access to the writeback features of AS, so you aren't limited to the strange spreading mechanisms that the Planning addin forced you to use.

I can’t wait to hear about partner’s using PivotTable writeback.

I'm still trying to decide what sessions to go to. As a developer I don't get as much customer interaction as the PM side (or sales & support), so I'm leaning towards trying to attend as many of the customer/partner presentations on PPS as possible as well as the PPS chalk talks that should be less PowerPoint driven. In addition I will be a proctor during some of the Hands-on-Labs (HOLs).

I'm registered at msbiconference.com, if you search for "petereb" you can find my profile. (I guess even BI is social now) During this week I'll figure out what sessions I expect to be at, when I'll be in the HOLs, etc. And I'll add that to the profile in case you really want to stalk me and complain about some bug in the AddIn. If you want to influence where I go so we can meet or have some focused discussions, please write in my wall at msbiconference.com or comment here.

Anyway I'm very pleased to see that others in the blogosphere are doing a good job disseminating information about PerformancePoint. I just want to highlight one post in particular because it is exactly one of the topics I wanted to cover: Sacha Tomey's excellent write up a couple months ago of the new matrix option "Clear Changes After Workflow Action" that was added in SP1. Covers all the points I wanted to and all the relevant pictures. Thanks Sacha!

I have some details about the addin performance improvements http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=3445168&SiteID=17.

Stage 1 - Contributors choose a Submit option for their assignment.

At this point when the add-in shows a confirmation dialog the submission has been placed into the asynchronous queue. The time it takes for this will vary based on the network connection between the add-in and the front-end server that hosts the PPS Planning web service. The number of values and formulas entered (and if "Include Workbook" has been checked, the size of the workbook) determine the amount of data that needs to be transferred. (Technically so does the number of dimensions in the fact-table, but generally for scalability scenarios the number of rows is more important than a single row's size). At this point the assignment status will be "Pending".

Stage 2 - Planning process service processes the item in the queue

If the queue empties, then the planning process service will wait and check the queue again later. The amount of time it waits is called the “Poll interval (milliseconds)” under the “Planning Process Service” tab in the admin console:

If the queue contains something when one of the items is done being processed then it will immediately process the next available item. i.e. there is no waiting. So generally, for scalability, I've seen our server team recommending poll intervals of at least a minute (60000 ms). During periods of high submission frequency like the end of a budgeting cycle or some other deadline. The admin will have to balance the desire for near instantaneous processing with the database and server workload.

Obviously in addition to this poll interval the time to actually process the submission is necessary. This includes checking the submission for data validation problems or permission errors. Then deleting the old records in the fact table and adding new ones. As well the status and changelist needs to be updated for the assignment and other metadata. And of course the partition that the data was loaded into needs to be marked dirty so that the cube will know to read the data when it is reprocessing.

After all this the planning process can go back to the queue to look for more work or to enter a wait period... At this point the status for the assignment will be "WaitProcess".

Stage 3 - Cube reprocesses dirty partitions in the fact tables

A separate interval is defined in the admin console called “OLAP cube refresh inteval (seconds)” under the “Workflow” tab:

On this interval the partitions for each fact table will be checked. If any of them are dirty then the cube is told to reprocess itself. Analysis Services is pretty smart so it will only need to reread the data from dirty partitions in the fact tables. So this will limit the amount of time that it takes to transfer updated data from the SQL tables to the AS cubes. Then there is some calculation time in the cube before it can start publishing the new data. During processing the old data is still available, so there's no down time. Finally after this point, the assignments that were waiting for processing will be marked "Partial" or "Submitted" as needed...

Here there's not such a straightforward guideline for what to set the interval to. For small data submission volume a lower value in minutes or seconds might be ok. But for really really large data volumes where immediate access is not critical, some organizations might even set it to values in the hours. It all depends on how close to real-time the data is needed and whether or not the server hardware can really handle that kind of workload.

So does that explain why its hard to say exactly when your data will be available to others? Experimentation and a little patience are in order. Really once you've used PPS for a few cycles I think the time from submit to availability becomes pretty familiar.

A quick side note: an assignment cycle just blocks further additions to the queue. So for contributors who are really pushing the deadline, all they have to do is get the confirmation that their submission was accepted before the deadline. That way the deadline is still meaningful and fairly predictable to AddIn users.

Playing in the land of Peter Eb. while writing this post...Criminal by Fiona Apple from