NEricson's Weblog

Sample Warehouse Adapter

nericson — Fri, 02 Apr 2010 13:37:00 GMT

I have just finished a sample Warehouse Adapter that demonstrates how to customize the Warehouse and Analysis databases. This is a fairly simple sample, but should help out anyone considering customizing the Warehouse.

Take a look:

https://code.msdn.microsoft.com/Tfs2010SampleAdapter

TFS 2010 method to get Work Items as of a date

nericson — Mon, 08 Mar 2010 08:58:00 GMT

Quite a while ago I added a blog post about using compensating records to get Work Item data as of a specific data from the Warehouse.

TFS Warehouse Work Item Tracking Compensating Records

In TFS 2010 we have added a new field, Revised Date, that allows you to find the work item dimension data quickly. The way it works is that there are two dates associated with each Work Item dimension entry stored in the Warehouse. A "Changed Date" and a "Revised Date". I'll be the first to admit that their names don't really tell you much about what they are used for - hopefully this should clear things up a bit.

Changed Date (the DimWorkItem.System_ChangedDate field in the Warehouse) stores the date that that revision was created.

Revised Date (the DimWorkItem.System_RevisedDate field in the Warehouse) stores the date that the next revision of the Work Item was created. If there is no next version (this version is the latest) then this will store the date equal to CONVERT(DATETIME, '9999', 126) - or '9999-01-01 00:00:00.000'.

So if you want to get the Work Item revision for a specific Work Item ID that was the latest version as of '2010-01-04'

SELECT
wi.[System_Id]
,wi.[System_Rev]
,wi.[System_Title]
,wi.[System_State]
FROM
[DimWorkItem] wi
JOIN
[DimTeamProject] tpc
ON tpc.ProjectNodeSK = wi.TeamProjectCollectionSK
WHERE
wi.[System_ChangedDate] <= '2010-01-04'
AND wi.[System_RevisedDate] > '2010-01-04'
AND wi.[System_Id] = 1234
AND tpc.ProjectNodeName = 'VSTSDF'

You can see how I "box" the date I'm interested in with the Changed Date and Revised Date fields. Changed Date and Revised Date are full DateTime data types - so you have down to the ms granularity for these fields. So you could get the valid value as of, say, 5:00 PM on '2010-01-04' as well if you would like.

Just a couple notes on the query above - you will notice that I have added a join to DimTeamProject here as well. This is because Work Item ids are no longer unique across a TFS deployment. They are unique to a Team Project Collection (TPC). So in order to fully specify a Work Item I need to specify both its TPC and Id.

I hope this to be the first of several posts on the new schema within the Warehouse - if anyone has any specific things they would like to see let me know.

Previous State and State Change Count

nericson — Wed, 29 Apr 2009 14:58:00 GMT

Today I am going to look at two interesting fields in the Warehouse related to Work Items – Previous State, and State Change Count.

The Previous State field stores the state a Work Item was in before it entered its current state. This field is in the [Work Item] table in the relational/dimensional Warehouse and in the [Work Item] Dimension in the Cube.

The State Change Count measure counts the number of changes in State between a Work Item revision and its previous Revision (actually instead of 0 it will have a value of NULL – so it will either have a value of 1 or NULL). It is in the [Work Item History] table in the relational/dimensional Warehouse and in the [Work Item History] measure group in the Cube.

The easiest way to understand their values is with an example. In this example we are going to create a new Work Item and then edit it several times. The table shows the values in the Warehouse (of course all of these are not in the same table in the Warehouse – but showing it like this makes the example simpler).

1. Create a new Work Item.

ID	Rev	State	Previous State	State Change Count	Date
1	1	Active	NULL	1	1/1

The Previous State is NULL because prior to Revision 1 the Work Item had no state. The State Change Count is 1 because the Revision changed from state NULL to state Active.

2. Modify a field other than state (i.e. Title)

ID	Rev	State	Previous State	State Change Count	Date
1	1	Active	NULL	1	1/1
1	2	Active	NULL	NULL	1/2

The State the Work Item was in, before it was in its current state, is still NULL. The state of Revision 2 didn’t change from Revision 1 so the State Change Count is NULL.

3. Modify the state (i.e. Resolve the Work Item)

ID	Rev	State	Previous State	State Change Count	Date
1	1	Active	NULL	1	1/1
1	2	Active	NULL	NULL	1/2
1	3	Resolved	Active	1	1/3

Now the State the Work Item was in before it was in its current state is Active. The state changed so the State Change Count is 1.

4. Modify a field other than state (i.e. Title)

ID	Rev	State	Previous State	State Change Count	Date
1	1	Active	NULL	1	1/1
1	2	Active	NULL	NULL	1/2
1	3	Resolved	Active	1	1/3
1	4	Resolved	Active	NULL	1/4

5. Modify the state (i.e. Reactivate the bug)

ID	Rev	State	Previous State	State Change Count	Date
1	1	Active	NULL	1	1/1
1	2	Active	NULL	NULL	1/2
1	3	Resolved	Active	1	1/3
1	4	Resolved	Active	NULL	1/4
1	5	Active	Resolved	1	1/5

Alright I think it is clear what the values will be for these fields… now what good are they?

The Previous State serves as a cache of state the work item before it entered the current state. Without this one would have to walk back work item revisions until you found one that had a different state. The Previous State can be used to answer questions like:

For my Work Items that are currently Active, how many of them have been reactivated (Previous State = Resolved)? A large number of reactivated Work Items may indicate developers are not doing enough unit testing.

The State Change Count gives us a way to quickly identify revisions where the state changed. Using this along with the Previous State you can identify things like resolved rates per day or incoming rates per day. For example if you looked for Work Item revisions with State = Resolved, Previous State = Active, and State Change Count = 1 you would get those revisions that were resolved – and the dates. Note that if a work item was reactivated then resolved again you would get it back two times – this may or may not be what you want for your report.

TFS Warehouse Work Item Tracking Compensating Records

nericson — Fri, 20 Mar 2009 17:43:00 GMT

Compensating Records

What are compensating records?

First a little background. Whenever a work item is updated a new revision is created. These revisions are moved over to the Warehouse and stored in the various Work Item Dimension and Fact tables. Each revision will be in the Warehouse – however each date will not have a revision. Something like (ignoring actual schema for now):

ID	Revision	Date	Remaining Work
1	1	4/10/2009	50
1	2	4/15/2009	40
1	3	4/20/2009	30
1	4	4/25/2009	20

Now if you wanted to ask a question about work items on a specific date then one way you can answer your question is to get the latest revision of the work item created before that date.

For example say you want to know the Remaining Work for a Work item on 4/22/2009. You could get all the work items whose Date is <= 4/22/2009 (Revisions 1, 2, and 3); then from these find the one with the largest Revision (Revision 3). This gives you a Remaining Work of 30.

However this requires you to do a MAX in your query and doesn’t work particularly well in the Cube. So we have added compensating records. These records are primarily used for running sum computations. Each one effectively negates, or compensates, the previous revision of that work item.

For example, using the same data from above:

1. 4/10/2009 – Work Item 1 is added.

ID	Revision	Date	Remaining Work	Record Count	Is Compensating
1	1	4/10/2009	50	1	No

2. 4/15/2009 – Work Item 1 is revised.

ID	Revision	Date	Remaining Work	Record Count	Is Compensating
1	1	4/10/2009	50	1	No
1	1	4/15/2009	-50	-1	Yes
1	2	4/15/2009	40	1	No

3. 4/20/2009 and 4/25/2009 – Work Item 1 is revised.

ID	Revision	Date	Remaining Work	Record Count	Is Compensating
1	1	4/10/2009	50	1	No
1	1	4/15/2009	-50	-1	Yes
1	2	4/15/2009	40	1	No
1	2	4/20/2009	-40	-1	Yes
1	3	4/20/2009	30	1	No
1	3	4/25/2009	-30	-1	Yes
1	4	4/25/2009	20	1	No

Now you can answer the same question as above by getting all the revisions of work items older than 4/22/2009 (Revisions 1, 1’, 2, 2’, and 3) and summing their Remaining Work. This gives you a Remaining Work of (50 + -50 + 40 + -40 + 30) = 30. This should be faster than without the compensating records (and it makes the Cube happy!).

What about that Record Count column?

You can apply the same running sum logic to the Record Count to get the number of work items as of a date. So you could ask how many work items were there on 4/22/2009. Again we sum up the values for all rows with a Date <= 4/22/2009 and we get (1 + -1 + 1 + -1 + 1) = 1. Not terribly exciting with our data, but you can see how if we had more work items this could be useful.

Again – with TFS

Alright so let’s look at some example using the TFS 2008 schema. Say I want to get the number of Active and Resolved Work Items assigned to me as of a given date. First thing I need to know is what tables I should look at:

Info needed in query	Warehouse Table
Active and Resolved Work Items	[Work Item] filtered on [System_State]
Assigned to me	[Work Item History] or [Current Work Item] joined to [Person] on [Work Item History].[Person] = [Person].[__ID] or [Current Work Item].[Person] = [Person].[__ID]
Work Items as of a date	[Work Item History] filtered on [Date]

Putting this together I see I need [Work Item History], [Work Item], and [Person]. So let’s build the query:

SELECT wi.[System_State]

,SUM([Record Count]) AS [Count]

FROM [Work Item History] AS wih

LEFT JOIN [Work Item] AS wi

ON wi.[__ID] = wih.[Work Item]

LEFT JOIN [Person] AS p

ON p.[__ID] = wih.[Assigned To]

WHERE [Date] < CONVERT(DATETIME, '2009-03-19', 126) AND

p.[Person] = N'Nick Ericson' AND

wi.[System_State] IN ('Active', 'Resolved')

GROUP BY wi.[System_State]

System_State	Count
Resolved	2
Active	6

I can change the query slightly to get a better query plan (please send any further optimizations along – I’ll add them):

SELECT wi.[System_State]

,SUM([Record Count]) AS [Count]

FROM [Work Item History] AS wih

LEFT JOIN [Work Item] AS wi

ON wi.[__ID] = wih.[Work Item]

WHERE wih.[Assigned To] = (SELECT TOP 1 [__ID] FROM [Person] WHERE [Person] = N'Nick Ericson') AND

wih.[Date] < CONVERT(DATETIME, '2009-03-19', 126) AND

wi.[System_State] IN ('Active', 'Resolved')

GROUP BY wi.[System_State]

One thing I should note that can be confusing is that I am passing in 2009-03-19 00:00:00.000 for the Date. Most people would consider this not to be 19^th, but instead the 18^th (or so barely the 19^th that we shouldn’t use it for determining how many bugs were Active on the 19^th). However in the Warehouse we truncate the times off of the Date field. So if a work item was marked active at 6 PM on the 19^th the Date field in the Warehouse will have 2009-03-19 00:00:00.000. This is a long way of saying that 2009-03-19 is the 19^th, not the 18^th.

Alright getting the counts for someone on a given day was fairly straight forward. You may want to add in some logic to return Active and Resolved counts of 0 when the query returns NULLs – but this depends on how you will build your report.

Trending

How about a trend report? You basically need to execute the queries from the section above over a range of dates. We can use a Cross Apply here (again send along your optimized queries):

SELECT d.[Date] AS [Date], ca.*

FROM [Date] AS d

CROSS APPLY

(

SELECT wi.[System_State]

,SUM([Record Count]) AS [Count]

FROM [Work Item History] AS wih

LEFT JOIN [Work Item] AS wi

ON wi.[__ID] = wih.[Work Item]

WHERE wih.[Assigned To] = (SELECT TOP 1 [__ID] FROM [Person] WHERE [Person] = N'Nick Ericson') AND

wih.[Date] < d.[__ID] AND

wi.[System_State] IN ('Active', 'Resolved')

GROUP BY wi.[System_State]

) AS ca

WHERE d.[__ID] BETWEEN CONVERT(DATETIME, '2009-03-19', 126) AND CONVERT(DATETIME, '2009-03-21', 126)

Date	System_State	Count
3/19/2009	Active	6
3/19/2009	Resolved	2
3/20/2009	Active	7
3/20/2009	Resolved	2
3/21/2009	Active	7
3/21/2009	Resolved	2

Welcome

nericson — Wed, 11 Jun 2008 02:45:59 GMT

I am a Developer for Visual Studio Team Foundation Server at Microsoft on the Reporting or BII team (it depends who you ask) and thought I would start a blog. I know there are lots of questions about Reporting on TFS data out there - hopefully I can address some of them here.

TFS Reports using XML and XSL

nericson — Wed, 11 Jun 2008 02:40:07 GMT

There is a lot of ways to get data out of the TFS Reporting Warehouse and into a report. Things like Excel and Reporting Services RDL Reports work well for people. Typically people use these tools to pull data from the Cube and display it in a consumable format. Of course these aren’t the only way to get data and display. Here I will look at using the relational Warehouse as a source and use simple XSL transforms to pretty up the data. This method will not work for everyone – in my opinion it is really only suitable for fairly simple reports.

Here I will create a report that displays my current work items (task, bugs, etc…). This is something like a “what am I supposed to be doing” report.

First we need to understand how to get the data from the relational warehouse. I will not cover all the ins and outs of the warehouse schema here. If you are looking for further information on this front take a look at the MSDN documentation (https://msdn.microsoft.com/en-us/library/ms244691.aspx) describing the various tables (Facts, Dimensions, and Fact Links) and their columns. I will look at getting Work Item data so here are the relevant tables:

Person – People’s names, Email, etc…
Work Item – Work item dimension information. This includes things like the Work Item ID, Revision, Title, etc…
Current Work Item – Current Work Item Fact information. This includes fact information and links to the appropriate dimensions. We will join this to the Work Item and person table to get the data we need in our report.

To get our report data our query would look like (alright we don't need CreatedData, but I am going to leave it in there):

SELECT * FROM
(
SELECT wi.[System_ID]
,wi.[System_State]
,wi.[System_WorkItemType]
,wi.[System_Title]
,cwi.[System_CreatedDate]
,DATEDIFF(d, cwi.[System_CreatedDate], GETDATE()) AS [Age]
FROM [dbo].[Person] AS p
LEFT JOIN [dbo].[Current Work Item] AS cwi
ON cwi.[Assigned To] = p.[__ID]
LEFT JOIN [dbo].[Work Item] AS wi
ON cwi.[Work Item] = wi.[__ID]
WHERE p.[Person] = 'Nick Ericson' AND
wi.[System_State] <> 'Closed'
) AS WorkItem
ORDER BY WorkItem.[System_State], WorkItem.[System_CreatedDate]
FOR XML AUTO, ROOT('WorkItems')

Which results in XML that looks like:

…

So now we need to apply the XSLT to transform this to HTML. It is fairly simple to apply a transform via some managed code. One could use a web services to deliver the report to users with relatively little effort. Another method would be to return the XML to the browser along with the xml-stylesheet tag (i.e. the two lines below added to the head of the XML). This could also be done via a web service or ISAPI DLL. I will not go into the details of how this would work here - if there is interest let me know.

where WorkItems.xsl is your XSL file.

Whichever method you choose, once the XSL (which is included below) is applied you will get the following report:

Bugs

ID	State	Title	Age (days)
14	Active	Some new work item rev again and again	89
15	Active	new bug a b c d e	85
16	Active	Test work item 0 created: 3/19/2008 9:39:05 AM	83
17	Active	Test work item 1 created: 3/19/2008 9:39:05 AM	83
18	Active	Test work item 2 created: 3/19/2008 9:39:05 AM	83
19	Active	Test work item 3 created: 3/19/2008 9:39:05 AM	83
20	Active	Test work item 4 created: 3/19/2008 9:39:05 AM	83

Tasks

ID	State	Title	Age (days)
1	Active	Set up: Set Permissions	110
2	Active	Set up: Migrate Source Code	110
3	Active	Set up: Migrate Work Items	110
4	Active	Set up: Set Check-in Policies	110
5	Active	Set up: Configure Build	110
6	Active	Set up: Send Mail to Team Members and Stakeholders about installation and getting started	110
7	Active	Create Vision Statement	110
8	Active	Set up: Create Project Description on Team Project Portal	110
9	Active	Create Personas	110
10	Active	Define Iteration Length	110
11	Active	Create Test Approach Worksheet including Test Thresholds	110
12	Active	Brainstorm and Prioritize Story List	110
13	Active	Set up: Create Project Structure	110

Now this is a fairly simple example but you can see that you can develop some reports fairly quickly using this method.

Below is the XSL. It isn’t the prettiest example out there but it is fairly straight forward to follow:

https://www.w3.org/1999/XSL/Transform">

Bugs

                  background-color: #9999FF

ID		State		Title		Age (days)

Tasks

                    background-color: #9999FF

ID		State		Title		Age (days)