We have a production instance of SQL Server Reporting Services to support the reporting needs of the SQL Business Unit. Jaime Tarquino on my team now owns the server, and we ran into an issue a couple weeks ago I worked with him on solving. It's been a while since I blogged so I thought I would share this issue with you in hopes you can learn from it, and let you know that we in the dev team also understand the experience of dealing with production system issues that affect our business and our regularly scheduled work.

We came to work for a busy day of meetings and test stuff as usual, but on this day we started seeing emails from users saying "why did I get 8 copies of this report in email today?". Usually subscription problems manifest themselves by not getting delivered to people's InBoxes, now they were getting 8 (EIGHT!) versions of the same report. So we knew something new was up.

Our implementation of RS is configured as a 2-node NLB farm with the RS instances installed on separate machines, and the catalog database on a 3rd machine. We typically run the latest released version of RS on the box, or something close to it, which includes CTP, RTM, and SP releases of RS, so we upgrade the server several times per year. During such an upgrade we typically follow a process similar to:

• Make a backup of the RS catalog database and the encryption key from the production machine
• Restore the catalog database to SQL on another machine
• Install the original, production version of RS on the test machine and configure it to use that instance of SQL Server for the catalog, and restore the encryption key
• Verify a few reports and other features are working correctly
• Upgrade RS to the new version
• Verify a few reports and other features are working correctly
• If so, perform the upgrade process on the production instance

As we had just finished up the release of SQL Server 2012, we had followed these steps to upgrade our production server, but ran into an issue in step 5, "Upgrade RS to the new version". The old version of RS was an internal build and was only a minor version upgrade, so setup could not upgrade the instance, and we had to abort the upgrade process until the next week.

We figured our current problems were related to something that happened during the upgrade testing, and we started investigating the issue using some of the techniques I've covered in this blog. The first thing we noticed was that there were a lot (1000's) of subscriptions queued up in the Events table to be processed, and there were thousands of entries in the trace log file with this error:

"Schedules, events, and subscriptions processing may be delayed because processing queue reached the maximum item count = <nn>. Items currently being processed: <nn>."

We added the logging of this error in our previous release to help identify when the queue is not getting processed.

Both of these are a clear sign that something is not right, but it only explains why reports would not be delivered, it does not explain why 8 copies were being delivered. As an interim solution to avoid the "why am I not getting my report?" emails from our users, we finally deleted the older entries from the Events table, it was clear that the backup was so large that RS could not keep up with them, and it was even falling farther behind.

Along the way, we'd noticed a few other facts that did not seem relevant or related to the issues at the time:

• There were entries in the trace log file for successful subscriptions (based on their schedule IDs), but not all had the records in the execution log table we would expect from the execution of the subscription's report
• One of the RS server's trace log file had excessive numbers of "Logon failure: unknown user name or bad password" errors
• One of the rogue subscriptions had been disabled two weeks prior
• Of all the subscriptions (100's) on our production box, only a few were delivering extra copies of the report

Well into the evening Jaime noticed that one of our instances of RS was configured to use the instance of SQL Server with the backup copy of the catalog database, not the production catalog database. This was the smoking gun. We have a new guy in our IT department that owns and supports the systems, it was a common human error. Things started to fall into place; it was pretty clear that we had two instances of RS processing the same subscription in two different catalogs.

This would explain one extra version of the report being delivered, but not 8, and and why were the old subscriptions still being processed?

As it turns out, the backup of the catalog database used for testing was fairly old, which explains why the old subscriptions were still firing. And apparently SQL Agent had been running on the machine for a while when RS wasn't, which put a backlog of entries in the Events table that were not being processed by RS. And when the RS service using that database was started, it began processing the backlog Event table entries for old subscriptions.

Also, the reports from the rogue subscriptions were all using data sources with SQL Authentication and SQL logins that did not have expiring passwords. The rest of the subscription/report datasources used Windows authentication, and the passwords in those data sources had all expired. Which was probably a good thing for us in the RS team; otherwise there would have been literally hundreds of users in the SQL Business Unit and elsewhere in Microsoft getting peppered with old and duplicate email subscription deliveries.

The remaining issue was the cause of the backlog of unprocessed subscriptions in our production catalog database. To be honest, I'm not sure I completely understand the cause of that issue; we lost a lot of evidence when we manually deleted the Events table, so the rest here is mostly speculation and educated guesses that I'm confident can explain the issue.

We did recall that several of the entries in the Notification table - for subscriptions currently being processed - in the production catalog database had been there for hours. We've had problems with runaway queries in reports in the past, basically infinite loops in report dataset TSQL code, so several of the threads had been tied up for a long time. It was not likely related to the upgrade process.

Our RS installation normally uses and needs two instances of RS to keep up with both subscriptions and ad-hoc reports during busy parts of the day. A single instance can successfully struggle to keep up, but performance does suffer and this likely contributed to the problem.

For the subscriptions that were running simultaneously on both instances of RS, the instance of SQL Server processing the same dataset queries in parallel likely suffered from locking and contention and that caused those reports to take longer than usual in the subscription.

And finally, many of the queued entries in the Events table were from data driven subscriptions. Data driven subscriptions will put one entry in the Events table for each row in the DDS query, so we likely had some unlucky timing relative to a heavy DDS schedule.

So in case it's not obvious, the lesson we learned is that when you are using a parallel test system like we do, make sure that the production and test RS server instances are both configured to use the correct instance of SQL for the catalog database.

I hope you find this interesting and useful.

Thanks, Dean

This posting is provided "AS IS" with no warranties, and confers no rights.

To give you an overview, I am the Report Server test lead in the Reporting Services team, and I'm the primary contact for the Reporting Services Server that supports the SQL Business Unit here at Microsoft. If you are not already familar with the mechanics of how subscriptions work, please review my other "Troubleshooting Subscriptions" blog postings, the information will below will make much more sense.

Introduction

When I came to work one monday morning, I noticed that I didn't receive the reports I normall get every morning, not the best way to start the week, I knew something was up. I have a report I created to monitor subscriptions in my server similar to the info in my other blog postings, and it showed that there were 8,000 rows in the Event table in our ReportServer database. The first indication here was that SQL Server Agent was running correctly because it is the source of the 8k records, but since there were no rows in the Notifications table, I knew that Reporting Services was not processing them as fast as they were coming in. One of my developer peers was helping me with the problem, he looked into the trace log files and noticed there there were entries similar to:

w WARN: Schedules, events, and subscriptions processing may be delayed because processing queue reached the maximum item count = N. Items currently being processed:

This is a confirmation of the problem I'd suspected, but I belive this error is only logged with SQL Server 2008R2 and beyond.

Note: I use the term "subscription" here alone, there were also "cache refresh" Events in the table and in my queries below. These are from the scheduled shared dataset cache refresh feature in SQL Server 2008R2, but the process for identifying and solving them are the same so I'll just say "subscription" from here on out.

Background

There is a database application that is most commonly used by reports hosted in our report server, and many of those reports have one or more subscriptions to deliver reports via email. That database server had been recently upgraded (and caused all kinds of other problems:) so I was suspicious that it was the source of the problem here. But to narrow it down, we had to learn a bit more about what entries were in the Event table to definitively trace it down to the problem, and we used the first query below to see which reports were potentially causing the subscription problems. After running the reports manually, there were a few that never seemed to return, and they were early in the queue, so they were the smoking gun.

Ultimately, the problem did end up being related to the database upgrade. The database the reports were originally written against had numerous custom indexes added by my peers with great query optimization skills. The database upgrade involved low level table schema changes, there were views based on those tables that were used by the report queries, and many of the custom indexes were not recreated during the upgrade. Hence there were many report queries that were now running much slower, by an order of magnitude, after the upgrade, and this caused the log jam in our subscription processing. There were also many other reports not affected by the database upgrade, they were either unaffected by it, or they were written against another database, but they were caught up in the backlog of the subscriptions for the slow reports.

At this point, all of the available polling threads in the Report Server were busy processing the subscriptions based on the reports with slow queries. Even more events were getting loaded into the Event table faster than the Report Server could process them. It was clear that the report server would not be able to recover on its own, and it was only a matter of time before executive emails would start going around asking what was going on. So there were a few things that needed to be done to get the system back into production.

Steps to Recover

Disable the subscriptions causing the problems in the first place

With 8k Events waiting to be processed, adding more new Events to the table would not help. And even after the current subscription queuing problem was solved, the problem would only come back if the same subscriptions continued to fire. To get an idea which reports had the most slowdown after the upgrade I created a report based on the second query below to get the average execution time of the reports before and after the upgrade. Thankfully there were only 20 or so reports that were causing the worst problems, and quite a few of these turned out to be invalid or outdated due to people leaving the team and so on. For each of these, I went to the report in Report Manager, and either set the subscription to expire yesterday, or deleted them outright if they appeared to be old or invalid.

Clear out the backlog of queued events

Now that I was comfortable the offending report subscriptions were disabled, the Report Server still needed to crank through all of the queued events. I knew I had to purge them since many of the events were days old and would not provide value. Reporting Services does not have exposed features for this purpose, I'd have to roll up my sleeves and attack it at the database level.

Disclaimer: we do not officially support updating the Reporting Services tables mentioned below, do so at your own risk and only in extreme production stoppage situations like I describe here.

Now that I had a pretty good list of the offending reports/subscriptions, I manually deleted the rows for the slow reports from the Event table with a query similar to:

DELETE Event WHERE EventData = <guidvalue>

The rows that were associated with a subscription have the SubscriptionId uniqueidentifier value in the EventData column, and those associated with a scheduled cache refresh had the ReportId in the EventData column.

Now that I had deleted all of the Events from the offending subscriptions and reports, I removed the rows that had been entered prior to that morning with a query similar to:

DELETE Event WHERE TimeEntered < '9/20/2010 12:00:00'

This reduced the queued events to around 2,000 rows, and after the Reporting Services server was restarted to reset the polling threads, it only took about 30 minutes to process all those events.

Summary

As a reporting services administrator, I'm lucky to be in the product group that has the experts to back me up in cases like this; the diagnosis and solution above was reviewed and understood before I did it. This is also the first and only time I've ever modified the database directly myself. My hope in providing this to you is that it will help you get out of a similar situation, but I also hope it doesn't encourage you to make other types of updates to the data in your Reporting Services database, or apply this solution for creative custom features involving the Events table. Much of the data in the database is very inter-related, as the test lead of this area I guarantee you that we cannot test and verify that the Reporting Services server can handle all the various types of changes that can be made directly to the database tables. So please, be careful, know your queries, and run them at your own risk!

And finally, this experience did initiate a discussion between me and my dev/pm peers as to how we can improve the Reporting Services features to handle this better in the future.

I hope this helps, Thanks!

Dean

 Event Table Query

-- Get the queued events associated with subscriptions

select s.SubscriptionId, e.EventData, e.EventType, 

'Subscription' = s.Description, 'SubscriptionOwner'=uo.UserName, 

'ReportPath'= c.Path,'ReportModBy'=uc.UserName, 

'MinEntered'=min(dateadd(hh,DATEDIFF(hh,GETUTCDATE(),GETDATE()),e.TimeEntered)),

'EventCount'=COUNT(*)



from ReportServer.dbo.Event e

join ReportServer.dbo.Subscriptions s on s.SubscriptionID = e.EventData -- EventData has the subscription id

join ReportServer.dbo.Catalog c on c.ItemID = s.Report_OID

join ReportServer.dbo.Users uo on uo.UserID = s.OwnerID

join ReportServer.dbo.Users uc on uc.UserID = c.ModifiedByID



group by s.SubscriptionId, e.EventData, e.EventType, s.Description, uo.UserName, c.Path, uc.UserName



union all 



-- Get the queued events associated with cache refresh plans on reports

select 'SubscriptionId'=null, e.EventData, e.EventType,

'Subscription' = 'n/a', 'SubscriptionOwner'= 'n/a',

'ReportPath'= c.Path, 'ReportModBy'=uc.UserName, 

'MinEntered'=min(dateadd(hh,DATEDIFF(hh,GETUTCDATE(),GETDATE()),e.TimeEntered)),

'EventCount'=COUNT(*)



from ReportServer.dbo.Event e

join ReportServer.dbo.Catalog c on c.ItemID = e.EventData -- EventData has the report id

join ReportServer.dbo.Users uc on uc.UserID = c.ModifiedByID



group by e.EventData, e.EventType, c.Path, uc.UserName



order by EventCount desc

Query to show the reports whose execution was much slower after the upgrade

select

'ItemId'=post.ItemID, 'Path'=post.Path, 'LastModBy'=post.LastModBy, 

'preExecCount'=pre.ExecCount, 'postExecCount'=post.ExecCount, 

'preAvgData'=pre.AvgData, 'postAvgData'=post.AvgData,

'pcntDiff'= case when pre.AvgData = 0 then 0.0 else ((post.AvgData-pre.AvgData)*1.0)/(pre.AvgData*1.0) end

from

(

 select

 c.ItemID, c.Path, 'LastModBy'=um.UserName,



 'ExecCount'=COUNT(*),

 'MinData'=MIN(TimeDataRetrieval/1000), 

 'MaxData'=MAX(TimeDataRetrieval/1000), 

 'AvgData'=AVG(TimeDataRetrieval/1000)



 from ReportServer.dbo.ExecutionLogStorage el

 join Catalog c on c.ItemID = el.ReportID

 join Users um on um.UserID = c.ModifiedByID

 where 

 el.ReportAction = 1 -- Render

 and el.Source = 1 -- Live

 and el.TimeStart between '9/16/2010' and '9/25/2010'

 group by c.ItemID, 

 c.Path, 

 um.UserName



) as post



join 



(

 select  c.ItemID, c.Path, 'LastModBy'=um.UserName,

 'ExecCount'=COUNT(*),

 'MinData'=MIN(TimeDataRetrieval/1000), 

 'MaxData'=MAX(TimeDataRetrieval/1000), 

 'AvgData'=AVG(TimeDataRetrieval/1000)

 from ReportServer.dbo.ExecutionLogStorage el

 join Catalog c on c.ItemID = el.ReportID

 join Users um on um.UserID = c.ModifiedByID

 where 

 el.ReportAction = 1 -- Render

 and el.Source = 1 -- Live

 and el.TimeStart between '9/9/2010' and '9/15/2010'

 group by c.ItemID, 

 c.Path, 

 um.UserName



) as pre on pre.ItemID = post.ItemID

where case when pre.AvgData = 0 then 0.0 else ((post.AvgData-pre.AvgData)*1.0)/(pre.AvgData*1.0) end > 1.0

order by  case when pre.AvgData = 0 then 0.0 else ((post.AvgData-pre.AvgData)*1.0)/(pre.AvgData*1.0) end desc

First of all, I’ll give a quick review of the factors you need to consider when making report caching decisions.

Considerations for Caching Reports

How long does the report take to run?

Caching a long running report will allow the report to be rendered faster and give a better experience for the user, but it will not improve the time for reports that are already fast for live executions. Long running reports usually also have a high processing cost for both the report server that is processing the report, as well as the data source server that is executing the report’s queries. Considering this will allow you to determine if a report should be cached.

How “fresh” does the report data need to be?

Some reports, for example, those that reflect information in a fixed prior date range, will always return the exact same data every time the report queries are executed. These are perfect candidates for caching. Some reports, for example those showing current “status” information are only valid with the most current data available. These are the worst candidates for caching. Most other reports fall somewhere in the middle; the value and accuracy of the report is directly related to the age of the data, some latency is ok, but beyond a certain point it really should be refreshed. Considering this will also allow you to determine if a report should be cached, as well as how long the report should stay in the cache.

How often is the report used, and how many people use it?

If a report is used very frequently, the costs of caching the report will be justified, but if the report is not used very often, the cost of caching it will not have any return.

When do users commonly use the report?

Some reports are repeatedly used at predictable times of the day, week, month etc, for example, a morning report showing the activity for the previous day. Other reports see a flurry of activity at non-predictable times, for example, when your manager sends the group an email questioning the status of a large project. Considering this will allow you to determine how and when the report should be cached.

What parameter values are commonly used for the report?

If it is a report without any parameters, or users commonly choose the same parameters every time the report is rendered, the cached version will be valid for most requests. But if users commonly specify parameters that are not the same as those used in cached versions of the report, the cached version will not be used, and the report’s queries will be executed live.

Will users want to go back and see old versions of the report created at various points in the past?

This is common for daily, weekly, or monthly status reports where users want to see older historic versions of the report.

All of these options must be considered together to determine if the report should be cached and how it should be cached.

Types of Report  Caching

There are two main types of report caching. When you click on the "Properties" tab of a report in Report Manager and choose the “Execution” option on the left side of the page, you will be configuring “Execution snapshots” that will only be kept in the cache for a specified period of time. When you choose the “History” option, you will be configuring “History Snapshots” that will store several cached versions of the report that can be manually chosen from the “History” tab of Report Manager at some time in the future. History snapshots can also store the cached versions of Execution snapshots.

Execution Snapshot Configuration Options

There are two primary options for configuring “Execution Snapshot” report caching. You will see that by default, the “Always run this report with the most recent data” and “Do not cache temporary copies of this report” options are chosen by default. This of course means that the report’s data set queries will be executed each time the report is rendered. You may be able to avoid the execution time experience for the report user, as well as the reducing the processing cost for the report server and the database server the report is based on, by caching the report.

The first option is controlling how the caching is triggered. You can have the report cached when a user renders the report and a compatible cached version is not available. This is good for reports that are commonly used at unpredictable times of the day/week/month. The first person to run the report will see the time delay for the report queries to be executed live, but it will be cached for later users, and when activity slows down again, the report server and data source servers will not pay the price of executing and caching the report. This is a good option if you want to cache several versions of the report with different parameter values. This is the method that will be used when you choose one of the “Cache a temporary copy of the report…” options in Report Manager.

Or you can have the report server automatically cache a new version on a specified interval or schedule, so a reasonably fresh cached version is always available. This is the method that is used when you choose the “Render this report from a report execution snapshot” option in Report Manager. It is a good option if you know when and how often the report will probably be used, but if the report is not used, the cost of executing and caching the report will be unnecessarily spent for no benefit to report users. This option is only good if the report has no parameters, or all the parameters have defaults, because the report server cannot prepare a cached version of the report if it does not know what users will specify when they run the report. One precaution to be made here is that you need to consider how long the report takes to execute live when you are configuring how often it should be triggered. I mentioned in my blog post about report performance a story about a subscription that fired every 15 minutes for a report that ran for over an hour. This resulted in several executions running in parallel and slamming the SQL Server box that was executing the query. The same situation can happen with report snapshots, and the obvious suggestion is to make sure you don't initiate frequent snapshots for long running reports, or you will make enemies in dba and report server admin circles!

The second option is choosing how long the report will stay in the cache. You can have the cached version expired and removed at set interval or a scheduled basis, depending on when the age of the data in the report becomes outdated. This can be specified by choosing one of the “Cache a temporary copy of the report…” options in Report Manager.

History Snapshot Configuration Options 

The options for history snapshots are exposed via the History option of the Properties tab in Report Manager. They are pretty straight forward and similar to execution snapshots in many aspected; they can be created on a scheduled basis or by active live requests from execution snapshots, and they can be automatically be cleaned up on a scheduled basis. There are a couple key differences; they are exposed and visible for viewing by selecting them in the History tab of Report Manager, whereas execution snapshots are automatically used (or not) depending on if they exist or have compatible parameters. You can also manually generate them (assuming the parameters all have defaults) using the "New Snapshot" option on the History tab.

Report Design Considerations for Caching Reports

There are a couple considerations to keep in mind for cached reports. The first issue is that it is important for users to know that they are seeing a cached and potentially outdated view of the report. You can accomplish this by adding a textbox near the top of the report with an expression similar to:

="Report Executed at: " + Now().ToShortTimeString

As I’ve mentioned several times above, the cached version of the report is only valid if the parameter values of the cached report are the same as the values chosen by the person running the report. For this reason, if you want to take advantage of report caching, you should ensure that the report parameters have defaults that will be commonly used by the report users.

Pre-loading the Execution Snapshot Cache

One of the drawbacks of non-scheduled execution snapshots is that they are only cached after the report was run once, and the first person to run the live version of the report pays the price of building the cached version. One way to avoid this is to have the report executed via a subscription that fires on a scheduled basis, and its execution caches the report without a user having to pay the price. A static subscription can be used if there are no parameters, all the parameters have defaults, or a single set of parameters will fit the needs of most users. You can also use a data driven subscription with a query that returns multiple sets of parameters to cover the needs of various combinations of parameters.

Identifying Existing Reports that are Good Candidates for Caching

As I’ve mentioned in another blog posting, and Robert Bruckner covers extensively in his blog, the report server catalog database has a view called ExecutionLog2 that has entries for every report execution. You can use information in this view to identify the reports that are frequently executed live along with the parameter values that are specified, and are good candidates for caching based on the caching considerations listed above. I created a report to serve this purpose, and it is based on the query below.

select top 1000

'ReportName'=reverse(substring(reverse(el.ReportPath),1,charindex('/',reverse(el.ReportPath))-1)),

el.ReportPath, 'LastModBy'=u.UserName, 'Parameters'=convert(varchar(max),el.Parameters),

'TotDurationMin'=Sum(datediff(ss,el.TimeStart, el.TimeEnd)/60.0),

'AvgDurationMin'=Avg(datediff(ss,el.TimeStart, el.TimeEnd)/60.0),

'MinDurationMin'=Min(datediff(ss,el.TimeStart, el.TimeEnd)/60.0),

'MaxDurationMin'=Max(datediff(ss,el.TimeStart, el.TimeEnd)/60.0),

'UserCount'=COUNT(distinct el.UserName),

'ExecCount'=COUNT(*)

from            ReportServer.dbo.ExecutionLog2 el

left outer join ReportServer.dbo.Catalog           c  on c.Path = el.ReportPath

left outer join ReportServer.dbo.Users             u  on u.UserId = c.ModifiedByID

where 

-- report parameters

(@reportpath = 'ALL' or el.ReportPath = @reportpath)

and   el.TimeStart     < DATEADD(dd,@daycount,GetDate())

 

and el.Source        = 'Live' -- only show live executions, not cached executions

and   el.ReportAction  = 'Render' –- only include rendering operations

and   el.ReportPath   != 'Unknown' -- exclude reports that have been deleted after executing

and   el.Status        = 'rsSuccess' --

group by el.ReportPath, u.UserName,  convert(varchar(max),el.Parameters)

-- Only show the report executions that happen frequently, report parameter

having COUNT(*) > @minexeccount

order by Avg(datediff(ss,el.TimeStart, el.TimeEnd)/60.0) desc

 

By putting interactive sorting on the report’s table columns, I can look for the longer running reports, those that are executed frequently with a given set of parameters and so on. And I added drillthrough links that allow me to easily filter to just the data for a given report, as well as a MAILTO: URL that lets me easily contact the person who owns the report.

You can also use a similar query to monitor cached report activity. Reports that are executed live will have an ExecutionLog2 entry with RequestType=Interactive or Subscription, Report Action=Render, and Source=Live. Those that are rendered from an execution snapshot will have Source=Snapshot, and those from a history snapshot will have Source=History. If you have created scheduled execution or history snapshots for expensive reports, but don't see anyone actually using them, you may want to consider canceling the snapshot.

Internal Scheduling and Storage of Report Snapshots

Disclaimer: we do not officially support querying all of the Reporting Services tables mentioned below, the columns and information in them may change in the future. 

In my blog post about monitoring and troubleshooting subscriptions, I described how the scheduling of subscriptions is controlled by a SQL Agent Job that places an entry in the Events table, and it is moved to the Notifications table when the report server is processing the subscription. The same scheduling and delivery process is used by the schedules that refresh execution and history snapshots, with the exception that snapshots are stored in the catalog database rather than being sent to a file or email address. You can use the same basic techniques for monitoring, diagnosing, and troubleshooting snapshots.

I mentioned several times above that the benefits of caching does come with a cost. Part of this is the execution cost of the report's dataset queries in the datasource server and part of this is in the report processing cost in the report server. But it also has a processing cost of managing snapshot cache entries, and a storage cost for storing the cached version of the report in the report server catalog database.

The report snapshot cached entries are stored in the Snapshot  table with information regarding when the cached entry was created, when it will expire, the parameter hash values used to determine if the cached report is valid for a new user request and so on.

The actual cached reports are stored in the Chunkdata table, and large report snapshots can require significant storage. For example, the Chunkdata table in my report server has 258 cached snapshot entries, with about 85mb of total data in the Content column, a max size of 14mb, and an average size of 335kb. The server does not use snapshots particularly heavily, at least until I start asking people to use them:), but this can be a significant storage issue in the report server catalog database if you do cache a lot of large reports, so it is something worth montoring from time to time in your server.

I hope this information helps, take care, thanks, happy reporting, and of course:

"This blog is provided 'AS IS' with no warranties, and confers no rights."

Dean

For these reasons, I want to monitor the Report Server and know what is not working in the reports and subscriptions.

Let me start by giving a high level view of how subscriptions are defined and processed, but first:

Disclaimer: we do not officially support querying all of the Reporting Services tables mentioned below, the columns and information in them may change in the future.

Note: This information refers to the SQL 2008 catalog, this info will generally also apply to SQL 2005, but I've not verified it there yet. Also, unless otherwise specified, the tables mentioned below are in your Report Server catalog database with whatever name you gave it, and yes, you need permissions necessary to query them.

Ok, back to the fun stuff. 

 When you create a subscription several things are added to the RS server:

- A row is placed in the Subscriptions table identifying the name of the report, along with parameter settings, data driven query info and so on to process the subscription

- A row is placed in the Schedule and ReportSchedule tables with the timing of the subscription

- A SQL Server Agent job is created to control the scheduled execution of the report, and this is stored in the sysjobs and sysjobsteps of the MSDB database. The agent job name is a guid value that represents the ScheduleId (yes, in case you've not yet noticed, this makes your Agent Job list messy with a bunch of guids that make it hard to work with your non-subscription jobs, we are hearing you:).

This query shows how the subscription information is stored in the various related tables, along with the most useful information you will need to diagnose issues later:

 I use a similar query in a report with parameters to filter to certain reports, subscriptions, owners, and last_status values.

When the subscription runs several things happen:

- The SQL Server Agent job fires and puts a row in the Event table in the RS catalog with the settings necessary to process the subscription

- The RS server service has a limited number of threads (2 per CPU) that poll the Event table every few seconds looking for subscriptions to process

- When it finds an event, it puts a row in the Notifications table and starts processing the subscription, which includes executing the data driven subscription query (if present), processing the report, rendering it to the specified format (HTML, PDF, Excel etc), and delivering the final result to either an email address or a file share. Note that there will also be a row in the ExecutionLog table for the report execution.

This  query shows the subscription processing information that you can use to diagnose problems:

select

'Report' = c.Path,

'Subscription' = s.Description,

'SubscriptionOwner' = uo.UserName,

'SubscriptionModBy' = um.UserName,

'SubscriptionModDate' = s.ModifiedDate,

'ProcessStart' = dateadd(hh,DATEDIFF(hh,Getutcdate(),Getdate()),n.ProcessStart),

'NotificationEntered' = dateadd(hh,DATEDIFF(hh,Getutcdate(),Getdate()),n.NotificationEntered),

'ProcessAfter' = dateadd(hh,DATEDIFF(hh,Getutcdate(),Getdate()),n.ProcessAfter),

n.Attempt,

'SubscriptionLastRunTime' = dateadd(hh,DATEDIFF(hh,Getutcdate(),Getdate()),n.SubscriptionLastRunTime),

n.IsDataDriven,

'ProcessHeartbeat' = dateadd(hh,DATEDIFF(hh,Getutcdate(),Getdate()),n.ProcessHeartbeat),

n.Version,

n.SubscriptionID

from Notifications n

join Subscriptions s on n.SubscriptionID = s.SubscriptionID

join Catalog c on c.ItemID = n.ReportID

join Users uo on uo.UserID = s.OwnerID

join Users um on um.UserID = s.ModifiedByID

Common problems with subscription delivery: 

If your subscription emails are not being delivered to Inboxes across the company, there are several different things that can cause this:

- The SQL Server Agent job is not firing

This could be because the service is not running due to a password expiration or some other cause, and RS does not see anything to process. In this case, get the ScheduleId guid from the above query, find the job with that guid as its name, and check the job history to make sure the job fired. You will also see that the Event and Notifications tables are empty.

- RS is not processing the subscription events

The main issue here is that the RS service event processing threads are not processing the events. The main cause here is that the event processing threads are all tied up, and not processing the events as fast as they arrive. You will see that the agent jobs are firing, but the Events and Notifications tables have a large number of entries. The number of threads is fixed, you cannot increase it, so you need to figure out what they are processing and why they are not firing or keeping up. You can get the subscription and report information, and run the report or the data driven query manually to see if they are healthy. One thing I noticed is that a person had scheduled a report execution snapshot to fire every half hour, but the report took over an hour to process, so at any given time, several instances of the event were running at the same time occupying all of the threads, and nothing else was getting executed.

You may also notice that everything is healthy, but there are common hotspot times in the day (for example, 8am or midnight) when a lot of subscriptions are set to fire at the same time, and they may take a long time to complete, so the subscription processing cannot keep up and falls behind. In this case, you may encourage people to run them at different times if possible, or you could consider adding a second reporting server configured as a two node NLB cluster with the other. In this case, both servers will be polling the events table in the RS catalog and processing the events in parallel.

- SMTP is not delivering the emails

 This could be because the email address is bad, or there are problems with the SMTP configuration. I won't go into SMTP here, I'm a database guy, but RS needs it to be working to deliver the report. The Event and Notifications tables will be empty (RS has done its part), the LastStatus column of the Subscriptions table and the subscription status in Report Manager will show an error similar to "Failure sending mail: The user or group name 'domain\username' is not recognized. Mail will not be resent."

- The report or subscription is invalid

Again, this can usually be tested by running the report manually, but there are a myriad of causes that will also show in the subscription status. For example, the datasource may be invalid due to an expired password or an account for a person who has left the company, a subreport may have been removed or renamed, a server, database, or table used in the query may have been removed or renamed and so on. Statistically, this is the source of over 90+% of failed subscriptions I've seen.

So in summary, you can write reports based on the information in these catalog tables to monitor subscriptions, and take action before you get the "I've not received reports for days" emails, and encourage the report and subscription owners to keep their stuff clean and healthy and be good reporting citizens.

In case its not obvious, you may want to host these monitoring reports on a different server because if subscriptions are not firing, you may not get the subscription based monitoring report to tell you that subscriptions are broken!

I hope this helps, please let me know!

Thanks, Dean

As usual, "This blog is provided 'AS IS' with no warranties, and confers no rights."

Bear in mind that report execution time is directly related to the query execution time, the amount of data returned by the dataset query, and the complexity of the grouping, sorting, filtering, and aggregations in the report definition. Usually I can review the report, query, table design, or indexes and improve them dramatically. More often than not, the report writer does not have the option of changing table designs or indexes as they may not have admin or dbo permissions, or if they do, design changes would require broader application changes. So working on the dataset query and the report definition is your best bet.

To start, I usually extract the query from the report and run it outside the report to see what's going on, mainly, how long does it take to execute and how many rows are being returned. Often times, the report may take 20 minutes to run, and if the standalone query also takes 20 minutes, there is not much opportunity to address this in the report. I'll review the tables and indexes to make sure the queries are good from that perspective, or if the same results can be obtained more quickly with a different approach in the query. Application developers who work with the database regularly may also be able to help.

Another source of perf problems is contention, locking, and blocking in the database if there are a lot of users or processing using the same table. The resource manager in SQL Server Management Studio, 'SELECT * from SYSPROCESS', or SP_WHO will all show how many active processes are running on the box, and if there are blocked processes or large WAITs on lock resources. If there is, you may be able to benefit from breaking up the query.

For example, in our test automation application, there are typically hundreds of processes running concurrently in the database where the automation application is running tests, testers are verifying labruns, and overzealous report writers like me running expensive reports, and all of them are getting and holding locks on the tables throughout the life of the query. The application has one "hot" table that is used by practically every process, as well as 10 or so other tables that are commonly joined with the hot table in queries. I've found that if I get only the necessary information from the hot table with a well indexed query, load this into a #temp table, and then collect the remaining data from the other tables joined with the #temp table in separate queries, the overall data can be returned to the report much more quickly. Now my peers in the relational engine may cringe if they read this, because it is nearly always better to use SELECT only without #temp tables if possible, but when there are several tables and lock contention going on, this can help dramatically.

One of my first big wins in my reporting career (before I joined RS) was when my manager at the time had a labrun status report that took about 30 minutes to run. I was very familiar with the database, so he asked me to take a look and see if I could make it better. When I ran the query outside the report, it ran in a few minutes, but it returned about 20,000 rows. When I looked at the report, I saw that it was only displaying around 100 rows to the user, and I knew I was on to something. I noticed that the data was being displayed in a group section based on several columns in the report's dataset query, and the fields had several =COUNTROWS() and =SUM(columnname) aggregate functions in the report expressions. I took the columns used in the report table's group definition and added them to a GROUP BY clause in the query along with the other non-aggregated columns, and added the COUNT(*) and SUM(columnname) expressions to the column list of the query, and it executed in about 20 seconds! Then I removed the group from the table in the report, pasted the new query in the report dataset, and displayed the result set columns in the detail section of the table. I'd just converted a 30 minute report to a 20 second report, SCORE!!!

When Reporting Services was running the original report, it had to execute the query, cache the result set, build b-trees and calculate the aggregations, and finally, build and render the report. In the new version of the report much of the data processing was done by SQL Server; Reporting Services only had to execute the query, cache a much smaller dataset, and then build and render the final report, which by the way looked exactly like the original report. While we do everything possible to optimize Reporting Service's processing of data, SQL Server is optimized to this and can do it much more efficiently.

There is one flaw with this approach; report users commonly like to see the detail behind aggregates, so you can return the full data set and display it in a detail section that is hidden by default and toggled at the group level if the user wants to see it. To address this need, I still use the approach above, but provide a drillthrough link on the aggregated number to another report that shows the detail for the aggregated group.

Robert Bruckner covers some of this material in his blog post here, but one thing I'd like to repeat is that you can also use the ExecutionLog2 view in the Report Server catalog to identify and isolate these two issues. It has columns to show how much time is being spent in executing the report's dataset query vs the time spent processing the data. If you are a report server administrator, you can monitor the long running reports and provide my suggestions above to the report owner without going through the process yourself.

I hope this helps, please let me know!

As usual, "This blog is provided 'AS IS' with no warranties, and confers no rights."

I'm a true local, born and raised and educated in the Seattle area. I have a bachelor's degree in Business Management and more or less picked up computer skills along the way. My first real job after college was at a company called Microrim that shipped the first truly relational database product called R:Base System V, which was PC Magazine's Product of the Year around 1988 or so. I've been at Microsoft since 1990, and my first role was supporting Windows 3.0 during its first release. Around 1993 I joined the SQL Server product group as a software test engineer, and was the ONLY tester for the SQL Server Engine and the TSQL programming langauge (it takes about 300 people to fill that role now:), and shipped SQL Server 4.2, 6.0, and 6.5. Then I was a technical writer (gag) during SQL 7.0 owning half of the TSQL Reference and conceptual material. Then a tester in the Replication team during the creation of the Merge Replication feature set for the 8.0 release, and was the test lead when I left. Following that I was a test manager for the SQL "Integration Testing" team, a test lead for the Execution Engine through the 9.0 release. Since then I've been a test lead in the Reporting Services team, and own the "Server and Management Tools" components that includes the RS Windows and Web services, the SOAP and WMI APIs, the RS catalog, setup/upgrade, Report Manager, Config tool, Sharepoint Integration, and several other misc components and features.

To the core I'm a database guy and love working with relational database application develeopment and implementation, but my passion for the last 4 years or so is creating reports with Reporting Services. I'm intimately familiar with all of the databases that support the SQL Business unit, primarily the applications for tracking bugs and managing the test automation, and have written hundreds of reports to help us run the business and ship great software. I also own the production Reporting Services server that supports the business unit, and spend a lot of time monitoring the server and helping others in the unit with their reporting needs.

I have many very talented peers who have much more depth in the core Reporting Services technologies, and also love to blog, so I'd like to take a slightly differently approach than them. What I hope to share with you in this blogs is what I've learned creating reports, using the other RS features, managing a production server, and playing with the catalog.

Please feel free to contact me and give me feedback on my blog, other questions you have about these areas, and any disagreements, corrections, or enhancements to my postings.

Take care, thanks, and of course "This blog is provided 'AS IS' with no warranties, and confers no rights."