Some things you should know about the Usage DB ahead of time.

1) It is organized into many partitioned tables, one being active at a time and the partitions wrapping around and eventually overwriting one another. This would make accessing the data complicated, so there are views that make these partitioned tables go away. You access the data over views.

2) The number of partitioned tables can be configured, changing the amount of data and the length of time that it is retained. As I recall this can be done with a PowerShell command and results in all the current data being lost. 31 days is the max retention I believe. I should check all this …

3) The Usage DB tends to quickly become one of the biggest databases on the system, so many admins will want to turn it off, or reconfigure it to store only a small amount of data.

4) Querying the Usage DB can impact your running system, so if you really want to party on the data, pull a backup and restore it somewhere else and use that.

5) The Usage DB will have sensitive data in it including PII data since it records the urls, user login, etc. I suppose someone needs to write a cleanser for it.

Anyway, here is some code to access a single table (The RequestUsage View) in the Usage DB using Linq. I used Visual Studio 2008 for this. Have fun.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Data.Linq;

using System.Data.SqlClient;

namespace TestLinq

{

    class LINQQueryExpressions

    {

        [System.Data.Linq.Mapping.Table(Name = "RequestUsage")]

        public class SharePointRequest

        {

            [System.Data.Linq.Mapping.Column(Name = "PartitionId")]

            public byte PartitionId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "RowId")]

            public Guid RowId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "LogTime")]

            public DateTime LogTime { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "MachineName")]

            public string MachineName { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "FarmId")]

            public Guid FarmId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "SiteSubscriptionId")]

            public Guid SiteSubscriptionId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "UserLogin")]

            public string UserLogin { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "CorrelationId")]

            public Guid CorrelationId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "WebApplicationId")]

            public Guid WebApplicationId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "ServerUrl")]

            public string ServerUrl { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "SiteId")]

            public Guid SiteId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "SiteUrl")]

            public string SiteUrl { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "WebId")]

            public Guid WebId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "WebUrl")]

            public string WebUrl { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "DocumentPath")]

            public string DocumentPath { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "ContentTypeId")]

            public string ContentTypeId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "QueryString")]

            public string QueryString { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "BytesConsumed")]

            public int BytesConsumed { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "HttpStatus")]

            public short HttpStatus { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "SessionId")]

            public string SessionId { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "ReferrerUrl")]

            public string ReferrerUrl { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "ReferrerQueryString")]

            public string ReferrerQueryString { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "Browser")]

            public string Browser { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "UserAgent")]

            public string UserAgent { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "UserAddress")]

            public string UserAddress { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "RequestCount")]

            public short RequestCount { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "QueryCount")]

            public short QueryCount { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "QueryDurationSum")]

            public long QueryDurationSum { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "OperationCount")]

            public long OperationCount { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "Duration")]

            public long Duration { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "RequestType")]

            public string RequestType { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "Title")]

            public string Title { get; set; }

            [System.Data.Linq.Mapping.Column(Name = "RowCreatedTime")]

            public DateTime RowCreatedTime { get; set; }

        }

        static void Process()

        {

            // Specify the data source.

            string dbname = "mcm05_usagedb";

            string connectionString = "Data Source=(local);Initial Catalog="+

                                      dbname+";Integrated Security=SSPI;";

            var sqlcct = new SqlConnection(connectionString);

            sqlcct.Open();

            if (sqlcct.State != System.Data.ConnectionState.Open)

            {

                Console.WriteLine("Could not open - " +

                                   " Ver:"+sqlcct.ServerVersion+

                                   " State:" + sqlcct.State);

                Console.ReadLine();

                return;

            }

            var dc = new DataContext(sqlcct);

            var reqtable = dc.GetTable<SharePointRequest>();

            IQueryable<SharePointRequest> reqquery = from req in reqtable select req;

            // Execute the query.

            foreach (var r in reqquery)

            {

                Console.WriteLine("Part:"+r.PartitionId + " Row" + r.RowId+" "+ r.UserLogin);

            }

            Console.ReadLine();

        }      

        static void Main()

        {

            Process();

        }

    }

}

Measureable Demand Concepts

Precisely defining our concepts in measurable quantities is a difficult task and we do not pretend to have done it here. Each concept need a study in itself backed with supporting data to show that the measurement technique is successful. Investment (not a huge amount – but a significant amount) needs to be made to accomplish this.

What we really need is a demand model that derives from business drivers. Although we can use RPS as a measure of our demand (and we do in some later chapters), it is in no way a business driver and should be considered a temporary substitute. A business driver would be something that a BDM (Business Decision Maker) can provide technical assistance, like how many users need to use the system and what operations will they perform against it.
We will run through the major concepts we need here and list the alternatives we have for definition.

• Workload – While workload seems to be a well-defined concept, in fact it is a problematic one. If we plan for one workload and our deployment is in fact used for another then we will have a problem. A related question is when two different workloads can perhaps be considered to be the same from the point-of-view of resource consumption. A workable measureable definition could probably be obtained by looking at the incoming requests in the IIS logs and categorizing them. Although some work has been done by SharePoint test teams in this area, much more remains to be done.
• Total User Number – This is easy for most cases, namely the number of users in the user profile database. However for an anonymous situation it is more problematic since we do not have the users name and cannot actually distinguish between the users. Fortunately this is not a hugely important planning case.
• Active Users – This is a controversial – but probably necessary abstraction. The idea is that there are a certain number of users who are active – i.e. about to use SharePoint. An alternate formulation would be those that are deriving value from SharePoint at this instant. This is a somewhat smaller number than the Total User Number.
  Note that it is unclear how to measure this number – techniques range from installing user agents to register it directly, taking some fixed factor of the distinct users seen in logs a time period, or setting up an advanced Markov-process model and estimating the parameters numerically have all been suggested, but they all have obvious drawbacks.
  
• Concurrency – This is one of the toughest concepts to define. Architects frequently refer to “active users”, who are requesting pages, then reading them (i.e. experiencing think time). However it is not clear that this is a well-defined concept (see above) and it is in any case very difficult to measure. However if we insist on having something measurable there are a couple of usable alternatives:
• Requests in flight – This is interpreted as the number of users who have requested a page load and are waiting for it to complete. This is not easy to measure as one would think since IIS log resolution is only one second and many page loads return in less than one second. While this could be rectified with some kind of IIS filter but we do not know of anyone who has pursued this, but we mention it here for completeness.
  The actual numerical values of requests in flight tend to be very low, thus on a collaboration system serving 150k users one will probably only be serving no more than 30 users at a single time (and 300 Requests per second). So percent concurrency on this measurement will be small fractions of a percent.
• Distinct Users per unit time – while this is an attractive and fairly straight forward value to use, it has some problems. The major one is that the unit of time to use is not clear. It is clear that the longer the unit of time used, the bigger the measured value will be. One could hope that there would be a sudden discontinuity at some time which would reveal the time interval to use – but this is not the case; it is a smoothly continuously increasing function. None the less it seems to be the best value we have and is occasionally used by the more savvy planners. To our experience the best values (corresponding to our guidance) seem to be in the 3-5 minute range – i.e. Distinct Users per 3 minutes, or Distinct users per 5 minutes. Divide this by the total user number to get the “concurrency”.
  
• Request Rate – This is bound up very heavily with the way we measure concurrency, since concurrency times this number (presumably the average request rate) will give us our actual RPS (which is fixed).
  So for example if we use “distinct users seen in 3 minutes” for our concurrency value, we will get a different average request rate per user than if we decide to use “distinct users seen in 5 minutes” since the product of the N*C*R remains fixed.
  
• Peak RPS - While peak RPS seems clear, a little thought will reveal that the number is always bound to a definite time interval. Thus we have the Peak RPS (i.e. the max RPS for any one second), the Peak Average RPS for a minute, the Peak Average RPS for an hour etc. Analysis show that RPS tends to closely follow a normal distribution, thus an average and a standard deviation characterize these values fairly well.
  But which RPS should be planned for? Of course it depends on your taste, but since this follows a normal distribution you can use standard bell-shaped curves and use fairly simple statistics to calculate the amount of time your deployment will exceed whatever value you plan for. This approach will be expanded upon in the following.
  We believe a better approach is to abandon completely the idea of a peak and talk of distributions instead.
  
• User Operation – what is a real user operation – i.e. the smallest unit of business value to the customer? A page load is one alternative for this, however this is already somewhat far removed from an IIS request, so it is not clear how to measure it.
  Once common classification begin used in the PG refers to page loads as PLT1 (the first time a page is loaded on a client – with the client cache uncharged), PLT2 (second time) and others – and these all have rather different ratios of requests per usage operation – the values seem to vary strongly depending on the workload, client configuration, patch level, customization, etc, and it is very hard to measure at the server. However as a rule-of-thumb values between 5-1 and 20-1 tend to be used when absolutely necessary. Clearly more investigation needs be done here.

Note that some of these concepts were used in the RPS Equation mentioned above. A diagram showing a possible interpretation of the RPS equation and the relationships follows:

Although this may seem fairly clear, the exposition above should have made it clear that we have different ways to measure and interpret the values and arrows in this diagram resulting in disagreement in the field as to what is correct. Since there is no authoritative guidance in this area, the conflicting interpretations of all these concepts co-exist which adds to the confusion.

So our situation is some of these values can be measured with the help of log files, but some cannot. Rarely is this done in the field – clearly more research needs to be performed and published in order to enable the ultimate goal of being able to estimate usage directly from business drivers. The measurement process for each value needs to be documented, and perhaps software written and the product instrumented to actually make these measurements. It is a not inconsiderable amount of work.

However the current situation is not hopeless even without this investment. We have rules of thumb for many of these values and a good idea where the values need to lie. However the definitions need to be firmed up and agreed up – along with sanctioned measurement techniques. These can form the basis of a calibratable model.

This is well defined and relatively easy to measure, but it suffers from not being a business driver and is thus not exactly what we need.

We do not mention the economic cost and green cost measures (like TCO and the Carbon Footprint), but they must be questioned as well, i.e. people think they are well-defined and measurable (to judge by the way they are talked about), but are they really? ]]> https://docs.microsoft.com/archive/blogs/mike_wise/the-abyss Thu, 15 Apr 2010 01:16:47 GMT https://blogs.technet.microsoft.com/mike_wise/2010/04/15/the-abyss/ We use the following diagram to illustrate “The Abyss”:

An explanation follows – however the essential points are in the diagram.

It is an interesting fact that most people who deal with SharePoint Architecture in the field usually belong in one of two camps. There are the Planners – the people who design and build SharePoint deployments to satisfy business requirements. Then there are the Operationalists - who maintain, extend and troubleshoot SharePoint deployments. While there are indeed people who do both activities most concepts tools and guidance are directed at one or the other group, and we will show that even these people who perform both roles end up in a somewhat schizophrenic situation, since it is almost impossible to relate the two in our current state.

The Planner Side

It is tempting to invoke the classic theoretical/pragmatic dichotomy and say that the planners are being theoretical, and the operationalists are being practical about the same problem. However in fact the planners are being pragmatic as well. They are in the position of having to design a system to meet business requirements and are forced to use measures to estimate the demand. The most important quantities in this “demand model” are:

• The workloads (previously called scenario) for which the system will be used.
• The number of users using the system for a required workload.
• The pattern of concurrent usage by these users.
• The number of requests (operations of business value) that they will be required per user per unit time.

These values are supposed to be used in the well known RPS equation (about which I will blog later):

                N * C * R = RPS

Where:

• N = Number of active users of the system
• C = Peak concurrency rate
• R = Average Request Rate per User (LTHE classification - Light = 20 R/hour, Typical=36 R/hour, Heavy=60 R/hour, Extreme=120 R/hour)
• RPS = Peak request/sec rate

This equation was documented in various places. And the spirit of this equation is correct - the number of users, pattern of concurrent users, Transactions (which were sometimes documented to be operations of business value, and sometimes merely IIS requests of various nature), Transactions per Second are all closely-enough linked to business drivers. But the problems become apparent when a planner tries to actually fill out this equation.

However it is not at all clear what these values really mean and little agreement both inside and outside of Microsoft as to what they should mean. Some experts had their own conventions, but they where not built-in into any of our metrics, but different experts used different conventions which merely added to the confusion. Thus well-meaning planners who use this demand model were likely to use all kinds of values, and different planners can justify widely varying values based on their understanding of the meaning of the terms – and came up with similarly widely varying resource requirement estimates as a result.

Also since there are no clear-cut and accepted ways to measure these values on a real system, experience turned out to be of little help. So a planner who visits a system a year or two after deployment to check the validity of his or her assumptions will find it practically impossible to do so.

We should also mention that our now frustrated planner even finds it very difficult to use an existing functional system with a similar workload to help, as the values that are needed are simply not being measured – these include:

• What is the workload being served?
• How many users are active?
• What is the user concurrency?
• What and how many operations of business value are being executed?

The Operationalist Side

The Operationalist in their highest developed form is a person with experience on a wide range of SharePoint systems. They have extensive experience troubleshooting performance problems and usually have a tight community and share experiences and lessons with each other in an exemplary manner. More to the point - they know where bottlenecks have occurred in the past (and are thus are likely to re-occur) and have a suite of methods that they can use to check these.
It is an interesting and sad fact that they will rarely (never?) be interested in or even exposed to the original planning estimates and documents for the system. In fact they will just assume it is probably not useful or wrong (there are problems, right?) and go straight to the system and make measurements, frequently from performance counters that measure resource utilization of one sort or another. Eventually (hopefully) they will find the bottleneck and take remedial measures. They have a high success rate (eventually) and work closely with the product group (at least more closely than the planners to my experience) to ensure that this knowledge is preserved and propagated.

However it is not a really satisfactory state-of-affairs, since there is usually a lot of suffering that occurs before problem resolution that damages our reputation and slows adoption.

And since there is no systematic methodology to relate their measurements to the actual planning values (which are not well-defined or measurable anyway) it is unclear how this knowledge should feedback in any case.

Capacity Management vs. Capacity Planning

So this is “The Abyss”. We have our planners, who create designs based on estimates and hope to get things right, but the guidance does not change (even from version to version). And we have our Operationalists, who adapt to changing situations and fix problems as they occur, but have no way to feedback their experience into a model that can be used for planning. Note that the Microsoft field has people in both camps. And it is well-known that the Operationalists enjoy informing the Planners that they have had to clean up after their mess.

The way out of course is to design a “Capacity Management” practice, that can be used by both parties in the same way. Planners and Operationalists should be the same people speaking the same measurable language. But they cannot do it alone. We must give them a framework that satisfies both of their needs.

Have we done this for the new version – SharePoint 2010? There are some very positive developments in this direction. However it is hard – in fact very hard – to come up with a solid predictive model that can handle SharePoint with all its workloads and possible configurations. So I expect the Abyss will only go away slowly, but I expect it to be much narrower now with SharePoint 2010. ]]> https://docs.microsoft.com/archive/blogs/mike_wise/capacity-management-vs-capacity-planning Wed, 14 Apr 2010 04:02:00 GMT https://blogs.technet.microsoft.com/mike_wise/2010/04/14/capacity-management-vs-capacity-planning/ This is the first in a series of posts around a topic that is very important to me. I have published and presented most everything here in the last couple of years internally in a number of venues:

• At internal SharePoint PG presentations (originally) 
• As part of the MCM SharePoint 2009 Capacity Planning training (which I wrote and delivered)
• In an internal ThinkWeek 2009 paper
• As parts of various TechReadys (an internal TechEd like event)
• And other venues...

However I have never put it out in something that was available outside of Microsoft. Now that it seems to be catching on (internally with SharePoint at least), I think it is time I went "public". So this will be a series of posts around that, including the best material from the ThinkWeek paper and dozens of presentations. So here we go:

The Ideal

We want to contrast the concepts of “Capacity Planning” and “Capacity Management”. Capacity Planning is what you do up-front, however it has a “throwaway” flavor to it which (as we will see) is particularly prevalent in the SharePoint world. The idea that a “Capacity management” methodology be used for the entire lifecycle of a deployment is not a new idea, and is actually the methodology that is promoted by both MOF and ITIL the two most widely promoted and accepted operational methodologies (at least in the Microsoft world in which we move) – and in fact they are very similar since MOF was mostly inspired by ITIL.

Both MOF and ITIL require a continuously improving process for a given deployment. Both require that some kind of modeling be used at all stages in a process that continually runs to active an optimal deployment even in changing situations. A high-level diagram of this process taken from the MOF documentation is shown here:

There can be some debate as to what “Modeling” consists of, at the minimum we would expect some process that would allow us to vary our resource configuration and somehow predict what kind of performance we can expect.

So how would we expect this to work in a SharePoint deployment situation? We show a more detailed diagram here:

In more detail:

Modeling – Here we model our installation – presumably using the demand model described below. For example the SharePoint Capacity Planner (SPCP) uses this model. This will give us a prediction as to what kind of resources we will need to host this demand. The result of this step is a model that we can use in the Optimizing step to make decisions. Green considerations (like the carbon footprint) should be part of this model.

Optimizing – The optimizing step is similar to the modeling step. In this step various assumptions and alternatives are considered using our model. At the end we have a configuration (possibly just a change to an existing config) that we want to implement.

Change Initiation – Perhaps the word “deployment” would fit better here, however in this step the configurations that we decided in the preceding step are implemented. This can be a very complicated step but it is not in the scope of this document.

Monitoring – Once we are live we will start monitoring – collecting logs and performance counters, monitoring event logs, and other relevant artifacts.

Analysis – In the analysis phase we take the artifacts gathered in the Monitoring phase and analyze them to find problems and evaluate the quality of the deployment. At this point we will normally have at least some divergences from what our model predicted. So in the next step we will need to update and adjust our model to reflect this.

 Note that there is a key difference between the first time we hit the Modeling step, and the second time. In the first time we are operating with no existing deployment data and can use our model “as-is”. In the second and subsequent rounds we need to adjust the model to reflect reality.

The Reality

There is a SharePoint Capacity Planning (SPCP) model used in the field for SharePoint 2007 – and it is often used in the first cycle of planning. However the model has fundamental shortcomings when it is attempted to use it in more than one cycle of planning. If there are differences in what the model predicted and what we measured, then we need to adjust (or calibrate) the model. Some common reasons for this are:

1.       The user workload is different than anticipated.

2.       The hardware that is being used behaves differently than the model predicts.

3.       The actual transaction costs are different because of customization.

4.       The actual transaction costs are different because the product has evolved.

Only the first point here can be directly addressed by SPCP (assuming we know how to actually measure the workload). However the other three are fixed - SPCP does not make provisions for adjusting or calibrating those values and we cannot use the model in this cycle. So we end up with a truncated Capacity Management Cycle as follows:

The end result is that modeling is performed only in the “planning phase”, and this sets us up for the “The Abyss”, described in the following section. While we do have a continuingly improving process - our deployments get better over time - it is a truncated one, because the model must be left out of the cycle – the model’s predictions cannot improve to match our experience. And a particular insidious result is that our future planning does not benefit from this experience.

it is possible to design a model without these shortcomings. We will talk about how to do this in future posts.

 Notes:
  Most of this text was written in early 2009, and some things have changed since then:

• The importance of lifecycle capacity management is recognized by the PG and some new features in 2010 support this (partly).
• Additionally SPCP has been withdrawn from support, without a substitute. This is a shame, I think they should have fixed it instead of killing it. But it is an opportunity for someone maybe.
• Some people argue that this does not matter, that it is simply a starting point. However I think it does, and I explain why in the next post.

I have delivered some large Enterprise SharePoint designs, and developed and teach the MCM SharePoint Capacity Planning Course. I also have the certification myself – which was a lot of work but rewarding in the end.

So here we go.

This post is really more of a "test" post, I am experimenting a bit to see what kind of graphics I can use, how fast it updates, how amenable it is to changes, etc. So bear with me for a week or so while I figure things out. (I figured out that you pretty much have to use Live Writer to do anything decent)

Any way, so here are a couple pictures of me:

And here is a picture of something I programmed once – if you don’t know what it is – post a guess: