Processing 120K emails takes more than a day using EWS Microsoft.Exchange.WebServices.dll v2.2

Question

I am try to process 120K email with in 12 hours but end up with throttling error or the processing is very slow. Below is the sample code I use to get the EmailMessage and it was quite slow. Please advise.

var msgs = inbox.FindItems(new SearchFilter.SearchFilterCollection(LogicalOperator.And, new SearchFilter[] { new SearchFilter.IsEqualTo(EmailMessageSchema.IsRead, false) }), new ItemView(10));

                foreach (var msg in msgs)
                {
                    try
                    {
                        var itemPropertySet = new PropertySet(BasePropertySet.FirstClassProperties)
                        {
                            RequestedBodyType = BodyType.Text
                        };

                        var message = EmailMessage.Bind(_service, (EmailMessage.Bind(_service, msg.Id)).Id, itemPropertySet);
                   }
               }

Answer 1

The way your code is written isn't very efficient eg you doing one request for every Email which is why your going to get throttled.

Also your paging code isn't efficient either you can return up to 1000 items per page (not that you want to do that in this instance) but you should at least for this use propertyset for Id only, you can then process the items in batches using LoadPropertiesfromItems eg

ItemView view = new ItemView(100); 
view.PropertySet = new PropertySet(BasePropertySet.IdOnly);
var msgs = inbox.FindItems(new SearchFilter.SearchFilterCollection(LogicalOperator.And, new SearchFilter[] { new SearchFilter.IsEqualTo(EmailMessageSchema.IsRead, false) }), view);
var itemPropertySet = new PropertySet(BasePropertySet.FirstClassProperties)
                     {
                         RequestedBodyType = BodyType.Text
                     };
inbox.Service.LoadPropertiesForItems(msgs.Items, itemPropertySet );
foreach (var msg in msgs)
             {

}

with what your doing you can play with the batch size usually 50-100 is okay, your still most likely to get throttled but at a much lesser rate but your processing speed will be a lot quicker eg your current could would be currently making 11 separate requests to get 10 Items bodies this would make 2 request to get 50-100 items.

Also one other thing to note is that doing

RequestedBodyType = BodyType.Text

if the bodies of the messages aren't already stored as Text it will cause the Exchange Store to do a on the fly conversion from Html, which can also influence why your getting throttled because its costly in terms of performance, so if you can process the HTML bodies that will give much better performance because that is what the majority of email will be stored in natively.