Offline Address Book Full Download Fails if Hardware Load Balancer in use

Artículo
01/09/2013

I came across an interesting scenario relating to the use of a Hardware Load Balancer publishing Exchange Client Access Servers. This relates specifically to how CAS distributes OAB and how Outlook decides which files it needs to perform an OAB update.

First, let's walk through how it is supposed to work starting from a new Outlook Profile (if you already know the OAB download/update process, feel free to skip to the next section):

Outlook finds (via AutoDiscover) the url to use to connect to the OAB distribution server. It will look something like https://mail.contoso.com/OAB/\<guid of OAB>/
Outlook attempts to download the OAB.XML file at that URL. This is the "manifest" file that lists out the other files that Outlook will need to download.
Outlook attempts to download the "template" file for whatever language Outlook is running. For English, it finds the entry in the oab.xml that specifies langid 0409. The line in the OAB.xml looks something like this:
<Template seq='7' ver='7' size='6028' uncompressedsize='26470' SHA='df80ec818f23bb6b23aa55cbc49d45986604af34' langid='0409' type='windows'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-lng0409-7.lzx
Outlook attempts to download the template file 1efdeb0a-56fe-488d-aed7-b3d9179193a2-lng0409-7.lzx
Outlook finds the file name for the full OAB download in the manifest
<Full seq='7' ver='32' size='2408' uncompressedsize='11481' SHA='c3874445e4942eac788523db974e60a37463cabb'> 1efdeb0a-56fe-488d-aed7-b3d9179193a2-data-7.lzx </Full>
Outlook attempts to download the full OAB file 1efdeb0a-56fe-488d-aed7-b3d9179193a2-data-7.lzx

Now Outlook has an up-to-date OAB. Outlook and the Server track OAB status using a sequence number. In the above manifest, you can see the current sequence number of the OAB is 7.

Next time (say 24 hours later) Outlook goes to update OAB it goes through the following process.

Outlook finds (via AutoDiscover) the url to use to connect to the OAB distribution server. It will look something like https://mail.contoso.com/OAB/\<guid of OAB>/
Outlook attempts to download the OAB.XML file at that URL. This is the "manifest" file that lists out the other files that Outlook will need to download.
Outlook compares its own sequence number to that in the manifest. In our example, Outlook downloaded sequence 7. When looking at the manifest we can see that the server is up to sequence 9
<OAL id='1efdeb0a-56fe-488d-aed7-b3d9179193a2' dn='/' name='\Global Address List'>
<Full seq='9' ver='32' size='2422' uncompressedsize='11496' SHA='6852ea7fc58cfe2788766efbec61bc27cf62a96e'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-data-9.lzx
</Full>
Outlook looks in the manifest to see if there are differential files available to get from sequence 7 to sequence 9
<Diff seq='9' ver='32' size='122' uncompressedsize='11496' SHA='05eed595e0fdf88245a48d42d80431337dff36cc'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-binpatch-9.lzx
</Diff>
<Diff seq='8' ver='32' size='132' uncompressedsize='11494' SHA='4e0e62a0c3af3786310f9bd6a9b802879bcde07f'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-binpatch-8.lzx
</Diff>
They are there! Outlook grabs those 2 files and applies them in order. Outlook is now up-to-date at sequence 9.

However, if either of these items are true:

The diff files that Outlook needed to get current aren't available (they are kept for 30 days by default)
New profile

Outlook will fall back to a full download and follow the first procedure.

When good manifests go bad…

Now…on to the meat of the matter.

Consider this scenario:

Client Access web based services live behind a hardware load balancer.
Outlook decides it needs to do a full OAB download.

Outlook goes to the OAB URL and grabs the OAB.XML file. As laid out in the Full download procedure above, the first thing Outlook is going to look for is the template file.

And then…failure. If you initiated the download manually, you may see an 0x80200011 error code. You will also see an Event ID 27:

Log Name: Application

Source: Outlook

Date: 12/20/2012 1:20:01 PM

Event ID: 27

Task Category: None

Level: Warning

Keywords: Classic

User: N/A

Computer: workstation1.corp.contoso.com

Description:

OAB Download Failed. (Result code in event data).

Event Xml:

<Channel>Application</Channel>

<Computer>cas.corp.contoso.com</Computer>

</System>

<Data>OAB Download Failed. (Result code in event data).</Data>

</EventData>

</Event>

If you review Dave Goldman's excellent blog post on this subject (https://blogs.msdn.com/b/dgoldman/archive/2006/06/19/troubleshooting-offline-address-book-downloads-with-outlook-diagnostic-logging-_2800_event-id-27_2900_.aspx ) you will see that the binary data starts with 0C which indicates that Outlooks is doing a full download. You can also see Outlook's current OAB sequence number 0817 and the Server's current sequence number 0C17. To translate these hex sequence numbers into decimal, flip the bytes. The client translates to 5896 and the server to 5900 in this example.

Having said all of that, there is little indication as to why the download failed. To attempt to see what is really going on, we can use the bitsadmin tool. The Binary Intelligent Transfer agent is the code that performs the OAB download for Outlook and it is a built-in part of Windows.

C:\>bitsadmin /list /verbose

BITSADMIN version 2.0 [ 6.6.2600.2180 ]

BITS administration utility.

Listed 0 job(s).

Well, that is not very helpful…we need to run this tool at the moment of the failure.

Let's try this batch file:

@echo off

:work

bitsadmin /list /verbose >> c:\bitsoutput.txt

goto work

Looking over the output file that was created, we see Outlook grabbing the OAB.xml file successfully and then it moves on to download the template file:

BITSADMIN version 2.0 [ 6.6.2600.2180 ]

BITS administration utility.

GUID: {D5820AE4-24C8-4C43-8B1D-728E3C16CB4E} DISPLAY: Microsoft Outlook Offline Address Book

TYPE: DOWNLOAD STATE: CONNECTING OWNER: CORP\owells

PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / UNKNOWN

CREATION TIME: 1/8/2013 11:23:46 AM MODIFICATION TIME: 1/8/2013 11:23:47 AM

COMPLETION TIME: UNKNOWN ACL FLAGS:

NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3

RETRY DELAY: 600 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 0

PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL

DESCRIPTION: Microsoft Outlook Offline Address Book Template

JOB FILES:

0 / UNKNOWN WORKING https://mail.contoso.com/OAB/9165d096-ff22-4a79-ba2d-b47d8645e220/418771d0-6766-4c27-ab27-b6068ffb56a8-lng0409-5958.lzx -> C:\Documents and Settings\owells\Local Settings\Application Data\Microsoft\Outlook\tmplts.tm_

NOTIFICATION COMMAND LINE: none

Listed 1 job(s).

Great. You can see it connecting to grab the template file for English with the sequence number 5958.

Uh oh!

BITSADMIN version 2.0 [ 6.6.2600.2180 ]

BITS administration utility.

GUID: {D5820AE4-24C8-4C43-8B1D-728E3C16CB4E} DISPLAY: Microsoft Outlook Offline Address Book

TYPE: DOWNLOAD STATE: ERROR OWNER: CORP\owells

PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / UNKNOWN

CREATION TIME: 1/8/2013 11:23:46 AM MODIFICATION TIME: 1/8/2013 11:23:50 AM

COMPLETION TIME: UNKNOWN ACL FLAGS:

NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3

RETRY DELAY: 600 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 1

PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL

ERROR FILE: https://mail.contoso.com/OAB/9165d096-ff22-4a79-ba2d-b47d8645e220/418771d0-6766-4c27-ab27-b6068ffb56a8-lng0409-5958.lzx -> C:\Documents and Settings\owells\Local Settings\Application Data\Microsoft\Outlook\tmplts.tm_

ERROR CODE: 0x80190191 - The requested resource requires user authentication.

ERROR CONTEXT: 0x00000005 - The error occurred while the remote file was being processed.

DESCRIPTION: Microsoft Outlook Offline Address Book Template

JOB FILES:

NOTIFICATION COMMAND LINE: none

Listed 1 job(s).

Requires authentication??? I bet that is a bit of a misinterpretation of what happened. How about we go see what files are on the server? When we looked at the OAB directory (c:\program files\microsoft\exchange\v14\client access\oab) we see that there is a template file there…but it has a sequence number of 5961! Unlike the binpatch diff files, Exchange only keeps the latest template file. If we go looking for 5958 and the server has 5961, we will not find it. The above error really means "file not found".

Why the heck did we go looking for an old template file? Looking at the OAB.xml file on the server, it has the proper sequence number and filename for the template file.

Were we not looking at the current OAB.xml?

We tried opening OAB.xml in IE on the client machine. What the…why does this OAB.xml seem to indicate the 5958 is the current sequence number?

So, we now know what is not working…we can't go looking for old sequences…that won't work.

But, how did this happen?

The Answer (at long last):

The default configuration on many hardware load balancers is to cache some http content. We were looking at a cached copy of the oab.xml! Thanks for the help, Mr. Load Balancer!

The Solution:

Turn off http caching on the load balancer. You can either disable it for all Exchange Web services, the OAB vdir only, or just the OAB.xml file depending on your preference and your load balancers capabilities.

Comments

Anonymous
January 01, 2003
I just wanted to say thank you, this ended up being my issue (we're using F5 load balancers) and now that it's fixed a lot of my "weirdness" with OAB is resolved. I never could figure out why the OAB wouldn't update on the clients when it was frst updated on the server, but it was never an urgent issue and always sorted itself out. This time though there was some information published in the OAB that was a HR issue and needed to be removed ASAP so I had to dig into this fast. Excluding the OAB path and dropping the cache down to 60 seconds cleared this right up and everyone is now getting the updated OAB without issue. Thanks!
Anonymous
January 01, 2003
Looking at that binary value it says that the server and client have the same OAB sequence number (BE12 or 4798 in decimal). Verify that this is the case by looking on the CAS and client to see what sequence numbers are there. Not sure that your issue has the cause as I describe here.
Anonymous
March 04, 2013
The comment has been removed

Compartir a través de

Offline Address Book Full Download Fails if Hardware Load Balancer in use

When good manifests go bad…

The Answer (at long last):

The Solution:

Comments

Recursos adicionales