Thursday, 13 June 2013

Outlook 2013 Bug when using IMAP - Beware the Bandwidth Blowout!

We recently had issues with our work internet connection that meant sizeable amounts of data were being chewed up without our users doing anything in particular or untoward.

Using the bandwidth monitor utility Networx (http://www.softperfect.com/products/networx/) (which breaks accumulated traffic down by process), we specifically found that Outlook 2013 was downloading large amounts of data (40-50MB) - many (but not all) times that a "Send and Receive" synch was performed. This was using IMAP running against the open source DoveCot (http://www.dovecot.org/) as a mail server.

Now by default, this is set to every 30 minutes. If a machine is on 24 hours day, this can add up to 2.4GB per user - which clogs up your network and really adds up on anything but an unlimited plan!

Cracking open Wireshark (http://www.wireshark.org/), I turned off SSL to see what was going on in terms of the IMAP commands going down the wire and the full extent of the download. Nothing stuck out apart from the fact that setting the "Mail to keep offline setting" to 24 months as below performs an IMAP "UID SEARCH SINCE 13-Jun-2011" to filter results. This is as expected.


To my surprise, with the default "All" Setting for "Mail to keep offline", you can also see that on regular but apparently random occasions every single email (including ALL content and ALL binaries) was being downloaded every time from the mail server - regardless of whether Outlook already had a locally cached version or not. Sometimes, the download will only be 5MB per click of "Send and Receive All". I tried to reproduce it and it tends to happen when you save Drafts or send/receive large items in your inbox - but I wasn't able to isolate the issue through a process of elimination.

This may well be part of the OST corruption issues mentioned here - the IMAP issues in Outlook 2013 are apparently high on the MS internal support agenda:
http://productforums.google.com/forum/#!topic/gmail/9alMlaOkMC4

In terms of the mail server, the installed version of Dovecot also seems to support LIST-EXTENDED for special folder support introduced in Outlook 2013 - so that doesn't appear to be the problem (unless there is a variation on the standard between Outlook/the IMAP server - which is not impossible).

So no clear conclusions I'm afraid. The one thing that is clear is that there are problems with Outlook 2013 and IMAP - based on the large number of problems I've seen floating around the web. The above problem is just one example of these. I've also seen that Outlook 2013 sometimes doesn't download new emails at all (even though other clients such as Android download from the same mail server correctly.). In addition, when working with drafts and saving the draft email repeatedly, Outlook will not be able to save it and say that it must be saved as a new copy within your Drafts folder.

Apparently, a fix to some of these problems should be released around August 2013 (as per http://productforums.google.com/forum/#!topic/gmail/9alMlaOkMC4)

In the meantime, to fix the heavy bandwidth usage issue, I can only suggest:
  1. Roll back to Outlook 2010 if you're using IMAP . This seems to be a popular option - although you cannot have Outlook 2013 and 2010 running simultaneously (unlike other MS Office Products).  [Recommended - this worked in my situation and reduced synch bandwidth usage from 50MB to 850KB]
  2. Perform a full initial synchronisation with the default "All" setting. Then reduce your "Mail to keep offline" time window to a smaller figure after this initial synchronization (e.g. 1 month) 
  3. Delete your local OST file and set up email again. This may give you temporary relief from the problem - but it wasn't a permanent fix for me.
  4. Use yet another client like Thunderbird - although this apparently also has issues with non-appearing folders with some IMAP providers as well.
  5. Reduce your synchronisation frequency (to something less than 30 minute full synchs). 
  6. Update your mail server if possible to ensure full compliance with standards. This isn't an option for me. You should also confirm this with a pilot/Proof of Concept to confirm that it fixes your specific issue.
  7. Clean out/auto-archive your emails!

DDK

20 comments:

quark said...

Thank you for your suggestions, I'm fighting against this bug since January when my 10 GB/month mobile internet plan was completely swallowed by Outlook 2013.

Thomas in Oz said...

A few clients of ours are struggling with this exact problem, but don't have the option of rolling back to 2010.

Have you discovered a permanent fix since writing this?

quartermileburner said...

Thanks for posting. It has helped my sanity.

I finally looked into my bandwidth issues because I was having to move from unlimited to limited bandwidth and identified Outlook 2013 as the culprit using free or trial versions of Networx and NetBalancer.

I’m an Outlook 2013 home premium subscriber and I have 3 IMAP accounts set up whose data files total about 1.8 GB and a Gmail IMAP (700 MB) and Hotmail ActiveSync (120 MB) accounts. Nothing too unusual I would have though and I didn’t have any bandwidth issues on any of the previous 3 version of Outlook.

I had Outlook set up to send and receive every 2 minutes and it managed to consume 210 to 240 MB per HOUR! Even when there were no emails being received.

I also have no way of rolling back to a previous version so I called Microsoft who said that there is no issue and that I must contact my mail service provider or an issue with another device which checks my email. The guy said that this is the first time that he has ever heard of an issue like.

It’s all very frustrating, particularly as there doesn't appear to be any end in sight.

Tom McQuarrie said...

This!

This is EXACTLY the issue we're currently experiencing. We've been moving clients from pop to imap based accounts, and our bandwidth usage has skyrocketed. 1 client has about 2gb of mail in their account, about 60mb in their inbox. Watching the mail logs, they were doing 34mb of downloads every 5 minutes.

Just want to clarify, did reducing the offline setting to something less than ALL cause outlook to do a proper sync and only download new messages, or does it just mean it now downloads a subset of messages based on the date, but still downloads every single message regardless of status?

John in Brisbane said...
This comment has been removed by the author.
John in Brisbane said...

Oops previous typo.....

Me too.

I'm chewing through about 5Gb a day. My 25Gb/month Telstra plan isn't holding up well.

Is there a solution yet?

JC

Willie said...

Since you are running Dovecot, I think suggestion #6 is a little silly. Dovecot is just about as standard as you can get when it comes to IMAP.

Wifiuk said...

would be great if someone had an idea how to fix this?

Rick said...

Hah, IT guy here and me and my company have been scratching our heads how one of our clients went from - to over 50gb of data within a few days.

This, we have determined, is the culprit. Now that I've seen these posts i'm sure of it.

Sigh

dom said...

Same issue here - I went back to POP3 a week ago, bandwith issues solved (I had peaks up 15Gb / day on IMAP !!!)

Jonathan said...

We operate mailservers and noticed that IMAP usage on one client account went from 20Mb or so per day to 6.5Gb and now 8Gb per day. We run Dovecot and client has Outlook 2013 so this looks like the issue. Not sure what changed on the day bandwidth shot up. Client was operating Outlook 2013 before and we have not updated Dovecot.

quartermileburner said...

it looks like this issue has been fixed in the latest update - 15.0.4551.1005

Jonathan said...

Hi quartermileburner

Sounds you got some good news for us there. Do you have a reference or Change History link for the latest release 15.0.4551.1005 so we can see the issue has been addressed? I can't find any info on the release.

Thank you.

6572bcb4-7b8d-11e3-8024-000bcdcb2996 said...

You wrote tha the preblem may be fixed with Outlook version 15.0.4551.1005. Is that really sure. Have you alaysed it?
The huge data amound pisses me off. I would have no problem to change the client if there would be another Exchange active Sync Client for Windows 7.

vasudha dharani said...

Online Training
This year’s Technology online training classes for many students in the world look radically different from those in the past. Powered in part by the 21st Century Software Solutions—providing online trainings on all technologies by real time experts.
Online training is a precious and cost-effective method for continuous education and overall efficiency development that provides employees with well-rounded individuals and motivated employees dedicated to building the carrier.

Convenient - As employees struggle to balance the demands of work and home, Online Training allows them to learn from work, home and on the road. Relevant - Since course content includes the most current topics, Online Training ensures training is related to a person's career path. Instant - Delivery over the Internet enables Online Training to begin with just a few mouse clicks. Affordable - Cost savings is a key attraction as corporations save between 50-70% due to elimination of travel expenses according to Training journal. Fun - By providing a charming interactive environment with dynamic content, Online Training not only effectively keeps people up-to-date, but interested as well. Easy to Use - Open an Internet browser and employees are up and running quickly. ROI to the Max - Affordable online training can be up to 93% cheaper than instructor-led courses. Self Directed and suitable - Employees to learn at their own pace and maintain control of learning “where, when and how” with boundless access 24/7by 21st Century Software Solutions. contact@21cssindia.com
http://www.21cssindia.com/courses.html

Doug said...

Fought this problem for 3 days, I went in and increased the server timout period to 4min 20 secs in the Account Settings->More Settings->Advanced slider (nothing magic about that number I just grabbed the slider and moved it way right) and since then Outlook 2013 / IMAP on an Inbox with 7000 mails is flying - no repeat syncing - no huge bandwidth syncs.

Davids in Europe said...

Not sure if this will help, but I just went into Preferences/accounts/advanced/ and ticked 'download headers only' and 'except in inbox'.

Seems to have done the trick. Went from 500+kb/sec to 3kb/sec/

Hope it works for you too :)

Priece Rich said...

I fixed my problem with Gmail/Outlook 2013 sync by going to Google and managing my account.

There is a spot to turn on "Access for less secure apps." They don't recommend that you do it, but I'm now able to receive my Gmail after setting up a new account in Outlook 2013.

I hope this helps.

Sandman3

Manjit said...

Reducing your "Mail to keep offline" time window to 1 month seems to work

Devi.Angularjs said...

I have read your blog its very attractive and impressive. I like it your blog.

.Net Training in Chennai | .Net Online Training

Dot Net Online Training | LINQ Online Training ASP.NET Online Training