Managing the oversubscribed can be a challenge for many enterprise customers as Microsoft does not provide out of the box tools for monitoring and identifying the oversubscribed, much less managing the challenge itself.  This blog post provides an approach to identifying the degree of oversubscription, the users who are oversubscribed and who they are oversubscribed to, and a method for managing the challenge. 

Note:  The queries and sample code-sets provided in this document are not actual product code and are best tested first in a lab environment before being run against a production pool.  While the queries do not result in a change to the product, they will introduce an additional load to the backend database when run.  Likewise, since the SQL trigger offered as a potential solution was not released with the product, it is not supported code.

How MaxSubscription Works

When a user is oversubscribed (they are at the MaxSubscription limit), other users who attempt to add them to their contact list will not see Presence data.  However, the experience is somewhat inconsistent because the nature of “subscriptions” is dynamic and the number of subscriptions do not directly correlate to the number of users due to MPOP and modalities (for more information, see this 3^rd party blog: https://msunified.net/2010/10/25/contacts-vs-subscribers-in-ocs-2007-r2/ ). 

In short, there are at least five category subscriptions taken-out on a user for each person who has them on their contact list.  As such, 200 users=1000 subscriptions.

Consider the following scenario for “Alice”:

• Alice logs into Lync in the morning.  200 other users who have Alice on their contact list login between 9am and lunch.  Each user’s client takes five category subscriptions out on Alice: calendarData, contactCard, note, device, services, state.  At this point, Alice is “oversubscribed”.  Note that I stated each user's "client": if a user is logged into OCS R2 on multiple devices/endpoint (such as a Client Web Access and on a desktop client), then that user will take 10 subscriptions for Alice.
• As such, when Bob (who has Alice on his contact list) logs into Lync just before lunch, he sees no presence data for Alice, only a message that states: Max Followers Reached.  And then he calls the Help Desk - :)

However – Bob will not always have this experience.  For example:

• When everyone leaves for lunch (including Bob), half of the users don't just lock their machines but actually “logoff” - so their subscriptions to Alice are cancelled/cleared. 
• After lunch, Bob logs in before most of the other users do.  When he does, he will be able to take out a subscription on Alice and see her presence.  However, users who login after Alice is eventually oversubscribed later that afternoon will not - even if they could earlier that day.

Obviously, the nature of this dynamic can give a user community the impression that the product isn't working effectively, and while oversubscribed users are still accessible via IM - the lack of presence data is certainly a challenge to those who work consistently with that person. 

To eliminate the challenge, only two options exist:

1. The subscriptions taken-out against that user must be reduced – which requires the IT Administrator to either augment the subscription process or to simply augment contact lists.  The former requires a tool while the latter requires that the Administrator know who is oversubscribed and what users are regularly taking out subscriptions against them.
2. The limit for "subscriptions" be raised for certain users.  Microsoft does allow for the use of highly-targeted presence policies that enable this configurable limit of MaxCategorySubscriptions to be adjusted.  However there is no data published from Microsoft to indicate or calculate how the raising of this limit affects performance of the SQL backend.  As such, it is recommended to be used in either limited scope (few users) on a large pool or in a wider scope within a smaller pool (such as for executive-level users) where backend resources are not likely to be running near their limits (see more below).

Augmenting the Subscription Process via SQL Trigger

An unsupported sample code-set was developed for the Lync/OCS backend (a SQL trigger) by Microsoft engineer Tommy Mhire (tmhire).  This SQL trigger manages the subscription process for those who are already at 900 current subscriptions (though this value is configurable).  Our intent is that this sample code should be evaluated and monitored within the Innovate environment first and then gradually introduced to production pools.  This is a custom SQL trigger, and while not complex it is certainly not code from the Microsoft Product Group.  It is very likely that certain updates to the backend (Cumulative Updates) would result in a resetting of the RTCDYN SQL database which would require the re-application of the trigger, so seeing this as an add-on management “tool” is probably best until a better solution is found.

Below is some information on the trigger as well as the SQL code itself:

Behavior of the trigger

• Checks who the incoming category subscription is for then checks whether that publisher is over the trigger defined limit of category subscriptions (900)
• If the publisher is above the limit, it itemizes the subscribers who have category subscriptions for that publisher who ARE NOT on the publisher’s contact list
• It then take the subscriber with the oldest category subscription who is not on the publisher’s contact list and deletes ALL of their category subscriptions

Behavior in the environment

Subscribers who are on a publisher’s contact list WILL ALWAY see the publisher’s presence

• New subscribers ARE ALWAYS given precedence over old subscribers (i.e. old subscribers are pruned first)
• NO CHANGE to behavior is experienced if the publisher is below the category subscription limit

SQL Code

create trigger CategorySubscriptionFixer
on rtcdyn.dbo.CategorySubscription
FOR INSERT as
begin
                declare @PublisherId int
                set @PublisherId = (select top 1 PublisherId from inserted)
                if (select COUNT(*) from rtcdyn.dbo.CategorySubscription where PublisherId=@PublisherId) > 900 begin
                                declare @categorySubscriptionToDelete table (PublisherId int,DeliveryId int)
                                insert @categorySubscriptionToDelete
                                select top 1 categorySubscription.PublisherId,categorySubscription.DeliveryId
                                                from rtcdyn.dbo.CategorySubscription categorySubscription, rtcdyn.dbo.DeliveryContext deliveryContext
                                                where categorySubscription.PublisherId=@PublisherId
                                                and categorySubscription.DeliveryId=deliveryContext.DeliveryId
                                                and deliveryContext.SubscriberId not in (select BuddyId from rtc.dbo.Contact where OwnerId=@PublisherId)
                                delete from rtcdyn.dbo.CategorySubscription where
                                CategorySubscription.PublisherId = (select top 1 PublisherId from @categorySubscriptionToDelete)
                                and CategorySubscription.DeliveryId = (select top 1 DeliveryId from @categorySubscriptionToDelete)
                end 
end 

Risk / Impact of Use

• The trigger introduces minimal impact to SQL so any perf-related impact is seen as negligible at this time – to be validated during testing
• Subscribers who are not on the publisher’s contact list MAY BE pruned and left with a stale presence subscription. Given that any contact list falling into this scenario would have hundreds of subscriptions/contacts already, it seems very unlikely a user will ever be impacted by this since the most current subscriptions will be preserved.

Assessing the OverSubscribed

Some administrators may wish to understand the scope of this challenge for their environment before determining their next steps (such as adjusting their deployed topology or applying Presence Policies).  In my experience, even very large deployments (over 100,000 users) have only 1-2% of the population in this situation. The queries below provide several appraoches for estimating level of over-subscription.  Since both OCS R2 and Lync Server leverage a SQL backend and both manage the MaxSubscription and MaxCategory parameters the same way, we can leverage these queries against either backend to gather the relevant data. Once the data is gathered, the choice can be made to simply “reset” a user to zero subscriptions (affecting all subscribed users’ contact lists), to eliminate certain subscribers (affecting only their contact list), or apply a Presence Policy to the oversubscribed user.

Queries

Note: The following queries are not shipped with the product and thus are not officially supported.  Microsoft recommends testing these queries before use in production.  Special thanks goes - again - to Tommy Mhire for the work done to both draft and validate these for use with some of my customers.

Query 1: Identify the Oversubscribed

Use this to see which users are near the max value of category subscriptions (in the problem state or close to the problem state). Based on the default max of 1000 category subscriptions.

select Publishers.UserAtHost Publisher, SubscriberCountByPublisher.numSubscribers from (
     select PublisherId, count(*) numSubscribers from
     rtcdyn.dbo.CategorySubscription CatSubscription
     group by PublisherId
) as SubscriberCountByPublisher
inner join rtc.dbo.Resource Publishers on SubscriberCountByPublisher.PublisherId = Publishers.ResourceId
where numSubscribers > 990

Query 2: Identify Users Subscribed to a Specific Publisher

Use this to see all users subscribed to a publisher who's showing as "Maximum Followers Reached". Modify the "where UserResource.UserAtHost = 'user@domain.com' line.

select SubscriberResource.UserAtHost Subscriber, UserResource.UserAtHost Publisher, Categories.Name Category from
rtcdyn.dbo.CategorySubscription CatSubscription
inner join rtcdyn.dbo.DeliveryContext DeliveryContext on CatSubscription.DeliveryId = DeliveryContext.DeliveryId
inner join rtc.dbo.Resource UserResource on CatSubscription.PublisherId = UserResource.ResourceId
inner join rtc.dbo.Resource SubscriberResource on DeliveryContext.SubscriberId = SubscriberResource.ResourceId
inner join rtc.dbo.CategoryDef Categories on CatSubscription.CategoryId = Categories.CategoryId
where UserResource.UserAtHost = 'user@domain.com'
order by Subscriber

Query 1 and Query 2 are both effective for “inter-pool” subscriptions.  Since they look to the tables for the subscribed user, they provide data on subscribers homed on other pools.

Query 3: Identify all Subscriptions/Subscribers for an entire Pool

Use this to see all the category subscriptions on a given pool (Alice is subscribed to Bob three times).

select SubscriberResource.UserAtHost Subscriber, UserResource.UserAtHost Publisher, Categories.Name Category from
rtcdyn.dbo.CategorySubscription CatSubscription
inner join rtcdyn.dbo.DeliveryContext DeliveryContext on CatSubscription.DeliveryId = DeliveryContext.DeliveryId
inner join rtc.dbo.Resource UserResource on CatSubscription.PublisherId = UserResource.ResourceId
inner join rtc.dbo.Resource SubscriberResource on DeliveryContext.SubscriberId = SubscriberResource.ResourceId
inner join rtc.dbo.CategoryDef Categories on CatSubscription.CategoryId = Categories.CategoryId

Note: Query 3 does a combination of the work of the first two in that it will not only provide you a list of all subscriptions, but also the details about who the subscribers are.  As such, this query should be leveraged with great care as a live pool with significant load would certainly be impacted by running this query (as opposed to running Query 1 first, followed by a targeted search of subscribers for a given user (Query 2).

A Word about Presence Policies

Since OCS, Presence Policies have been available to administrators for managing subscription levels.  These parameters are referenced here: https://technet.microsoft.com/en-us/library/dd572188(office.13).aspx.  In Lync, these can be set via PowerShell (CSPresencePolicy). 

Two elements are detailed in the TechNet article: CategorySubscriptions and PromptedSubscribers.  One might read this and assume "prompting" implies the "asking of permission before you are granted a subscription", but that is simply not the case.  When you are prompted you are prompted, when you are not you are not.  Nothing about permitting or denying a subscription is within the control of a given user.

As such, the time may come where extending the presence policy for a specific population becomes necessary.  I would encourage great care in applying these policies.  Some blogs/posts I have seen reference a global setting for this - but there is absolutely no data to suggest what happens when these levels are raised.  Subscription processing is on the SQL backend, and for large pools raising these levels simply introduces extra load to the backend on what may or may not be an already busy pool.  Care must be exercised in such a way that the backend is monitored when this is done for pools homing any great population.  It may be better to re-home users who need a larger subscription base to a smaller pool with more available resources.  In that case, an entire pool can be set for a higher than 1000 MaxCategorySubscriptions limit - but that pool may not have more than hundreds of such high-profile users.

https://www.microsoft.com/downloads/en/details.aspx?FamilyID=27cfbbb9-73c8-4490-b7a4-31319863ebca&displaylang=en

But what constitutes health?  For OCS R2, a variety of technologies meet with base OS and hardware to provide the functions of even simple IM and Presence.  To that end, a fairly thorough combination of Monitoring (watching that certain metrics stay within predefined thresholds) and performance counter logging are required to both ensure maximum up-time and a conclusive investigation of root cause for service interruption.

Monitoring - OCS Front Ends and the SCOM Management Pack for R2

While it would be nice to have an out of the box, end-all be-all solution, that is not likely.  That said, the Management Pack for R2 provides some baseline information that can inform you of the health of your FE and its sense of connectivity to the world around it – particularly the BE (SQL). 

For example, with regard to connectivity to the SQL BE, a couple of key Front End metrics should be understood:

Queue Depth – Average number of database requests waiting to be executed

Queue Latency – Average time (msec) a request is held in the database queue

These two metrics speak to the dialogue between the OCS FE and the SQL BE.  Queue Depth is a measurement of “how big the line is”  for messages waiting to go to SQL, and the Latency metric tells us “how slow the line is” moving to SQL.  Both must be watched because a “big line” that moves fast is indicative of different issues than a line (big or small) that’s not moving at all. 

More importantly, if either of these reach a suboptimal level, users will begin to receive SIP 503 messages (OCS Server Not Available).  As such, alerting for these metrics is highly valuable as the opportunity to catch a problem before it impacts the user community is dependent on your ability to quickly address the root cause of the excessive queuing (SQL Disk I/O, memory shortage, SQL blocking, network, etc.).

Logging – What’s not in the SCOM Management Pack

Determining root cause of an OCS outage is no minor feat, and with the SQL BE being a point of consideration the surface area for a root cause search is rather large. 

Unfortunately, the OCS Management Pack is a bit slim on metrics enabled by default – and some are not even present that would absolutely be required for a correlation of data to, say, a SQL Diag report.  This correlation of data would allow an engineer troubleshooting an outage to pinpoint the source of the event more accurately because both the FE and BE are being logged thoroughly.

A list of minimum performance counters for troubleshooting performance on an OCS R2 FE is below:

\AVMCU - 00 - Operations\*
\AVMCU - 01 - Planning\*
\AVMCU - 02 - Informational\*
\CAA - 00 - Operations\*
\CAA - 01 - Planning\*
\CAS - 00 - Planning\*
\CAS - 01 - Informational\*
\LC:Arch Agent - 00 - MSMQ\*
\LC:AsMcu - 00 - AsMcu Conferences\*
\LC:CCS - 00 - TPCP Processing\*
\LC:ClientVersionFilter - 00 - Client Version Filter\*
\LC:DATAMCU - 00 - DataMCU Conferences\*
\LC:DLX - 00 - Address Book and Distribution List Expansion\*
\LC:IIMFILTER - 00 - Intelligent IM Filter\*
\LC:ImMcu - 00 - IMMcu Conferences\*
\LC:LDM - 00 - LDM Memory\*
\LC:QMSAgent - 00 - QoEMonitoringServerAgent\*
\LC:RGS - 00 - Response Group Service Hosting\*
\LC:RGS - 01 - Response Group Service Match Making\*
\LC:RGS - 02 - Response Group Service Workflow\*
\LC:RoutingApps - 00 - UC Routing Applications\*
\LC:SIP - 01 - Peers(_Total)\*
\LC:SIP - 02 - Protocol\*
\LC:SIP - 03 - Requests\*
\LC:SIP - 04 - Responses\*
\LC:SIP - 05 - Routing\*
\LC:SIP - 07 - Load Management\*
\LC:SipEps - 00 - Sip Dialogs(_Total)\*
\LC:SipEps - 01 - SipEps Transactions(_Total)\*
\LC:SipEps - 02 - SipEps Connections(_Total)\*
\LC:SipEps - 03 - SipEps Incoming Messages(_Total)\*
\LC:SipEps - 04 - SipEps Outgoing Messages(_Total)\*
\LC:USrv - 00 - DBStore\*
\LC:USrv - 04 - Rich presence subscribe SQL calls\*
\LC:USrv - 05 - Rich presence service SQL calls\*
\MEDIA - 00 - Operations(*)\*
\MEDIA - 01 - Planning(*)\*
\MEDIA - 02 - Informational(*)\*
\Memory\*
\Network Interface(*)\*
\Process(ASMCUSvc)\*
\Process(ABServer)\*
\Process(AVMCUSvc)\*
\Process(DataMCUSvc)\*
\Process(IMMcuSvc)\*
\Process(QmsAgentSvc)\*
\Process(OcsAppServerHost)\*
\Process(OcsAppServerHost#1)\*
\Process(OcsAppServerHost#2)\*
\Process(OcsAppServerHost#3)\*
\Process(smlogsvc)\*
\Process(RtcHost)\*
\Process(RTCSrv)\*
\Processor(*)\*
\TCPv4\*
\UDPv4\*

In the cases where there is an asterisk (‘*’) there is, within the SCOM MP, more than one counter in that subcategory.  For example, for the single entry, \AVMCU - 00 - Operations\* , there are seven actual counters:

As such, this “list” is far more comprehensive than just the 60 or so metrics given above.  In fact, the Management Pack has over 300 targets. 

However, it is important to know that over 60 of the targets in the MP are not enabled by default – and another 18 targets/categories mentioned above are not included in the MP at all:

\LC:Arch Agent - 00 - MSMQ\*
\LC:AsMcu - 00 - AsMcu Conferences\*
\LC:CCS - 00 - TPCP Processing\*
\LC:ClientVersionFilter - 00 - Client Version Filter\*
IIMFILTER - 00 - Intelligent IM Filter
LDM - 00 - LDM Memory
RGS - 00 - Response Group Service Hosting
RGS - 02 - Response Group Service Workflow
LC:SIP - 03 – Requests
LC:SIP - 05 – Routing
\LC:SipEps - 00 - Sip Dialogs(_Total)\*
\LC:SipEps - 01 - SipEps Transactions(_Total)\*
\LC:SipEps - 02 - SipEps Connections(_Total)\*
\LC:SipEps - 03 - SipEps Incoming Messages(_Total)\*
\LC:SipEps - 04 - SipEps Outgoing Messages(_Total)\*
LC:USrv - 04 - Rich presence subscribe SQL calls
LC:USrv - 05 - Rich presence service SQL calls
MEDIA - 02 – Informational

In order to gather adequate data for determining the root cause of intermittent service interruption or any other challenge with OCS FE health, these counters must be added to a perf logging utility, be it SCOM or otherwise.  Only then can the data collected be used in conjunction with SQL Diag and Perf data to isolate systemic failures.

Conclusion

OCS is a massive product with connectivity and dependencies to a variety of sources.  To  ensure maximum uptime and your best chance of conclusively troubleshooting an outage, it is recommended to go “above and beyond” standard metrics and – in some cases – the OCS R2 Management Pack for SCOM.  Organizations with the flexibility may wish to have two levels of logging available: one that is consistent in normal performance, and one that is triggered during performance degradation. 

______________________________________________

A Side Note about SQL

When there is a serious concern about OCS, contact a SQL support engineer for assistance in gathering a SQL Diag report and additional performance metrics there.  Typically, this Diag report includes:

Processor - %Processor Time - _Total
Process - %Processor Time – sqlservr
PhysicalDisk - % Idle Time <All instances>
PhysicalDisk – Avg. Disk Queue Length <All instances>

…as well as metrics such as deadlock tracing (also capable of being captured by turning on traceflag 1204).  For more on deadlock tracing, see: https://blogs.msdn.com/bartd/archive/2006/09/09/Deadlock-Troubleshooting_2C00_-Part-1.aspx. 

Typically, it is best to get a DBA to fine tune the DIAG as the capturing of Profiler data can consume a great deal of storage and may need to be run separately from the rest of the DIAG.

The questions, then, are how do you know who's using App Sharing to initiate sharing, who's using it to view only, and who's not using it all?

The answer comes to us in the form of the OCS 2007 R2 CDR and Monitoring Server.  This server role (previously two separate servers in the RTM version of OCS 2007) actually contains usage data on nearly all SIP traffic seen at the Front End servers.

When a client initiates an Application Sharing request, the SDP (Sessions Description Protocol) information within the SIP invite contains two required values: one that describes the type of event and one that defines the role for the person sending the invite.  These values are seen as follows:

In this case, it is clear that the type of the event we're looking at is for Application Sharing, and the user whose SIP URI is specified in the SIP invite is the actual organizer (sharer) for this event.

While default CDR reporting does not provide any data on Application Sharing metrics, the data is within the SQL database and can be retrieved using custom scripts.  Examples of such scripts are provided here, courtesy of Nick Smith at the OCS Center of Excellence:

To retrieve data on number of Multi-Party Application Sharing sessions by Organizer:

select      U.UserUri, Count(UserUri) as NumberOfApplicationSharingConferences

from  LcsCDR.dbo.Mcus as M,

            LcsCDR.dbo.McuJoinsAndLeaves as MJAL,

            LcsCDR.dbo.Conferences as C,

            LcsCDR.dbo.Users as U

where M.McuId=MJAL.McuId and

            MJAL.SessionIdTime = C.SessionIdTime and

            U.UserId = C.OrganizerId and

            MJAL.UserId = C.OrganizerId and

            M.McuType = 'applicationsharing' and

            MJAL.UserJoinTime is not null

group by U.UserUri

To retrieve data on number of Peer-to-Peer Application Sharing sessions by Organizer:

select      U.UserUri, count(UserUri) as NumberOfP2PApplicationShares

from  (select SessionIdTime,MediaId from LcsCDR.dbo.Media group by SessionIdTime, MediaId) as M,

            LcsCDR.dbo.MediaList as ML,

            LcsCDR.dbo.SessionDetails as SD,

            LcsCDR.dbo.Users as U

where M.MediaId=ML.MediaId and

            M.SessionIdTime = SD.SessionIdTime and

            SD.User1Id = U.UserId and

            ML.Media = 'Application Sharing'

group by U.UserUri

When leveraged, the results from the above queries would be reported as follows:

Marc