Share via

But I Really Need To Open 500 PSTs!

  • Article

Today we’re revisiting an issue I first encountered in Outlook 2003. We had a customer, a law firm, who kept all the mail associated with a particular case in a PST. Their users would pass these PSTs around, and depending on their case load, a single user might have over 100 PSTs open at the same time. This worked fine when they had Outlook 2000, but when they installed Outlook 2003 they found if they tried to open more than 20 or so PSTs, they started getting errors such as “Outlook failed to add the personal store to the session” and "Out of memory or system resources. Close some windows or programs and try again."

Although we don’t recommend opening so many PSTs at the same time, we investigated to find out what had changed, and whether we could help them out. In order to understand what had changed, it helps to know a bit about how the PST provider, and by proxy, Outlook, manages open PSTs.

There are two mechanisms involved in the PST provider’s management of PST files to consider. The first is the per-PST cache. Any time you’re dealing with random reads/writes to a file, maintaining an in memory cache is critical to performance. The PST provider uses an algorithm based on the file size and other internal details about the PST to determine how large of an in-memory cache to allocate. In Outlook 2002, we greatly increased the size of the caches we use for all types of PST files. Larger blocks of memory mean we consume more of the address space available to the process. In practical terms, this meant we could open about 20 average sized PSTs before we were no longer able to allocate a block of memory large enough to hold the cache for the next file. As a test, I tried to open brand new PSTs and was able to get 85 of them before we ran out of memory.

Based on our investigation, and the needs of this and other customers, we decided to go in and fine tune the cache sizes to allow more PSTs to be opened at once. This involved a tradeoff in performance, so we placed this change behind a registry key. The KB article 907651 documents the hotfix we generated for Outlook 2003, as well as the UseLegacyCacheSize registry key. Outlook 2007 includes the fine tuned cache as well, but the path to the registry key changes slightly. Whereas in Outlook 2003 the key is under “HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\ Outlook\PST”, in Outlook 2007 you’ll set it under “HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\ Outlook\PST”. (Incidentally, this fix was backported to Outlook XP too. There, the reg key will have a 10.0 in it.)

Now, once this registry key is set, if we try and stress test how many PSTs we can open, we’ll see we hit a practical limit of just under 300, but, unlike before, our address space isn’t exhausted. And where before, if your MAPI based process could only open 20 PSTs, you could always run another process and open 20 more, this new upper limit is spread across all MAPI processes on the box. This is due to the second mechanism the PST provider uses, shared memory.

As you might recall from my discussion of PST files in multiple profiles, the PST provider uses shared memory to track and coordinate access to PSTs across multiple processes. Each PST opened incurs a tax on this shared memory, and after enough PSTs are opened, there is no longer space in this block to track any more PSTs. As part of the work we did in introducing the UseLegacyCacheSize, we also increased the size of this shared memory block, which is what will allow you to reach nearly 300 open PSTs across multiple processes.

Now – just because you can open nearly 300 PSTs doesn’t mean you should. You’ll start to see noticeable performance issues when you get over 100 PSTs. In fact, if shouldn’t set UseLegacyCacheSize at all if you don’t need it. The smaller cache has a performance penalty. If you have an application which for some reason needs to open a large number of PSTs, I would consider designing it not to need all of them open at the same time. Only open the PST when you need it and unload it immediately after.

Comments

  • Anonymous
    June 10, 2009
    The post could not have been more timely. :)  Thanks Steve!

  • Anonymous
    June 11, 2009
    I am troubleshooting a system with this exact problem.  Office 2003 SP3 is installed, but I see no PST key under HKEY_CURRENT_USERSoftwareMicrosoftOffice11.0Outlook. On my own system, using Outlook 2007 SP2, I see no PST key under HKEY_CURRENT_USERSoftwareMicrosoftOffice12.0Outlook. Should I just create the PST key manually?

  • Anonymous
    June 11, 2009
    Yes.

  • Anonymous
    June 11, 2009
    Steve, Any guess much PSTs per profile Outlook 2010 x64 will support? Is there any info? That can be the way out for such situations.

  • Anonymous
    July 21, 2009
    Hi - I added the PST key as described; but am still getting the error.  I'm using Outlook 2003 on a newly-created account, and trying to add another .pst file; but it's falling over when I try to create the file, before I try to import anything to go into it.  Can't believe it's really a problem of size!

  • Anonymous
    July 21, 2009
    You don't say how many psts you already have open, or how large they are. If this is one of the first psts added to the profile you're not hitting the problem I described.

  • Anonymous
    July 24, 2009
    The comment has been removed

  • Anonymous
    August 10, 2009
    Just to add to the mix... What if you just have two (only 2!) mail stores you want to open simultaneously yet Outlook (2007) still runs out of memory?  My primary mail store is server-based ~235MB in size and I have a PST-based archive (mail older than 6 months) that is much larger - ~950MB in size.  Using two different profiles (one with just the primary store, the other with just the archive store), Outlook works just fine.  If I try to open both my primary and archive stores simultaneously however, I receive the "Out of memory" error.  I suspect the root cause of the problem here is that I am running Outlook with Business Contact Manager 2007 installed. My machine configuration is currently Vista-64 with 8GB RAM although I have also tried XP with 4GB RAM and Windows 7 RC with 8GB RAM - same problem.

  • Anonymous
    August 10, 2009
    Lloyd - whatever you're running in to, it's not this problem. And I doubt anything's running out of memory. Failures to create objects tend to get bubbled up as memory failures even if memory had nothing to do with the cause. If you think BCM may be part of the problem, try uninstalling it to determine if the behavior changes.