Wednesday 1 April 2015

Outlook 2013, Exchange Online and Office 365 - performance issues

Network latency, server response and DNS lookup:

• Use Outlook connection status to observe Outlook latency

Avg Resp: total latency
Avg Proc: server response time
RTT = Avg Resp - Avg Proc

Outlook Performance Troubleshooting including Office 365

• Use traceroute, pathping and psping to observe the routing path between Outlook and Exchange Online, network latency and packet loss.

How to measure the Network Round Trip Time to Office 365
Outlook 2010 - network latency test results

• Use ping outlook.office365.com to see what region the request is resolved in. Microsoft says that the request gets directed to the region where the DNS query is resolved, that is, it is advisable to use local DNS servers (e.g. your local ISP DNS) instead of global DNS servers (e.g. Google Public DNS or OpenDNS).

The servers are named by the regions: ApacSouth and ApacEast for South-East Asia, EmeaWest and EmeaEast for Europe, Middle East and Africa, NAmWest and NAmEast for US etc.

If necessary, as a temp solution configure a conditional forwarder that resolves outlook.office365.com to outlook-your.region.office365.com. You can also use a local public DNS service that correctly resolves outlook.365.com requests.

Office 365 users experience performance issues in Outlook when you use a public DNS service
DNS geolocation for Office 365, connecting you to your nearest Datacenter for the fastest connectivity
Office 365 client connectivity

• TCP Window Scaling – ensure the TCP window scaling was not disabled along the route.

• Packet loss - ensure packet loss and retransmission is around or less than 1%.

• DNS query latency – ensure the DNS query latency is in the range of 1 digit ms.

• MTU fragmentation – ensure packets are not fragmented along the route (optimal size 1460 bytes).

Office 365 Performance Management: (10) Troubleshooting the Top Customer Performance Issues

• Microsoft serves Office 365 customers through Content Delivery Network with entry points along the backbone. These entry points are routers that can be identified by the names ending in NTWK.MSN.NET. The first msn.net router that appears in a tracert report is the entry point to the MS network.

Office 365 Network Performance Troubleshooting
Microsoft Network Monitor 3.4
Microsoft Message Analyzer

• Use Microsoft Office 365 Datacentre map to understand where the clients are served and what route the traffic is traversing.

Office 365 Performance Management: (02) Office 365 Datacenters and Global Customer Network

Outlook and anti-virus scanners:

• Ensure Outlook files are not scanned by file-level anti-virus software.

• Ensure Windows Search files are not scanned by file-level anti-virus software since these generate and store Outlook index used by Fast Search.

• Ensure the exclusions are really working – test it using EICAR virus file.

• If you are scanning emails with an SMTP filter and have a file-level anti-virus scanner on clients, consider disabling email scanning and removing the AV add-in from Outlook.

Email Scanning can become redundant depending on your email environment. The On-Access Scanner in VSE scans attachments when running and/or writing to local disk. This happens after Outlook Scan finishes scanning. If email is scanned at the Gateway with McAfee GroupShield / McAfee WebShield, you may want to stop using Outlook Scan because of scan redundancy. (McAfee Technical Articles ID: KB52786)

If your antivirus software includes integration with Outlook, you may experience performance issues in Outlook. In this case, you can disable all Outlook integration within the antivirus software. Or, you can disable any antivirus software add-ins that are installed in Outlook. Be aware that if you are connecting to an Exchange Server mailbox, your mailbox or your email messages are already being scanned by antivirus software on the server. You should check with the Exchange administrator to make sure that this is the case. (Microsoft KBA 2695805)

How to remove or prevent installation of the VirusScan Enterprise On Delivery Email Scanner
Intended use EICAR - European Expert Group for IT-Security
Plan antivirus scanning for Outlook 2010
Supported Microsoft Outlook mail messaging architectures with VirusScan Enterprise 8.x

Outlook client-side performance:

• Keep Office up to date – especially with Service Packs.

• Ensure there are no unnecessary or incompatible add-ins in Outlook.

• Delete and recreate the OST file if it may be corrupted.

• Ensure the indexing is not stuck and the index database is not corrupted – rebuild the Outlook index if necessary.

Outlook OST file size guidelines:

• Up to 5 gigabytes (GB): This file size should provide a good user experience on most hardware.

• Between 5 and 10 GB: This file size is typically hardware dependent. Therefore, if you have a fast hard disk and lots of RAM, your experience will be better. However, slower hard disk drives, such as drives that are typically found on portable computers or early-generation solid-state drives (SSDs), experience some application pauses when the drives respond.

• More than 10 GB: When the .ost file reaches this size, short pauses begin to occur on most hardware.

• Very large (25 GB or larger): An .ost file of this size increases the frequency of short pauses, especially while you are downloading new email messages. However, you can use Send/Receive groups to manually sync your mail. For more information about Send/Receive groups, see the "Are you synchronizing many RSS feeds?" section.

How to troubleshoot performance issues in Outlook 2010
"Outlook not responding" error or Outlook freezes when you open a file or send mail
Outlook not responding, stopped working, freezes or hangs
Outlook performance is slow in the Office 365 environment

An issue introduced with SP1 for Office 2013 that affects Outlook calendar:

Log Name: Application
Source: Outlook
Event ID: 25
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Description: Could not read the calendar. Network problems are preventing connection to Microsoft Exchange.

Outlook errors when you try to access a shared calendar
Hotfix KB 2889951 for Outlook 2013 October 14, 2014 (Outlook-x-none.msp)

Outlook caching:

• Enable Outlook caching for primary mailbox

• Investigate pros and cons of caching non-email content

• Consider disabling Outlook hybrid cache mode (Exchange Fast Access)

• Consider disabling Outlook slow link detection

Outlook 2013 includes “hybrid mode”, which means that it’s got the ability to switch between cached and online data to display information to the user faster. The gate is 400ms, measured when the user logs on and connects to Exchange and updated when the user switches folders. If the network connection is good enough, Outlook can switch into hybrid mode to fetch data from the server and if not, access the OST.

Under some circumstances, the network adapter speed might not accurately reflect data throughput for users. For example, if a user's computer is connected to a local area network (LAN) for fast access to local file servers, the network adapter speed is reported as fast because the user is connected to a LAN. However, the user's access to other locations on an organization's network that include the Exchange Server computer might use a slow link, such as an ISDN connection. For such a scenario, where users' actual data throughput is slow, even though their network adapters report a fast connection, you can configure an option to change or lock down the behavior of Outlook. You can do this, for example, by disabling automatic switching to downloading only headers by using the Group Policy option, Disallow On Slow Connections Only Download Headers. Similarly, there might be connections that Outlook has determined are slow but which provide high data throughput to users. In this case, you can also disable automatic switching to downloading only headers.

When you add additional mailboxes to your Outlook profile and enable caching of the shared mailbox folders, Outlook individually registers every folder in the cached mailbox. Each of these registered folders counts toward the objtFolder type limit on the Exchange Server. As stated in the 9646 event that is shown in the "Symptoms" section of this article, the default limit for objtFolder objects it 500. If you cache one or more shared folders that contain several hundred folders, this limit can be exhausted. When this occurs, you may experience the problems described in the "Symptoms" section of this article. Additionally, a large number of Public Folder favorites can also contribute to exhausting the objtFolder object limit.

Configure Cached Exchange Mode in Outlook 2013
Only a subset of your Exchange mailbox items are synchronized in Outlook 2013
Outlook 2013 introduces hybrid cached mode
Performance problems when you try to access folders in a secondary mailbox in Outlook
Plan a Cached Exchange Mode deployment in Outlook 2013
Turn on or off Cached Exchange Mode

Firewalls and proxies

• Ensure the firewall is not terminating Outlook TCP Idle sessions too agressively.

• Consider either modifying the timeout of an idle session on the firewall or changing the default value of TCP KeepAliveTime (2 hours by default).

• Ensure the firewall has sufficient capacity to handle additional sessions.

Rather than being transient, Outlook connecting to Exchange (be it on-prem or Cloud based) opens up TCP connections and leaves them open for the length of time the application is open.

Under most circumstances, these connections will see traffic on very regular intervals and thus any idle timeouts won't be an issue. However, it is a fact, and one I've seen occur many times, that Outlook, if not performing any actions, may not send any traffic on an open TCP connection for a long period of time.

We saw this issue regularly when On-Prem was prevalent. Firewalls would kill idle TCP sessions after a period of time, causing disconnect pop-ups in Outlook, hangs or other problems within the application such as password prompts as it reconnected. These were often due to the firewall not informing the client of the disconnect by sending a reset. Thus when the client tried to use the connection again, it would send a packet, get no response, then retransmit five times, exponentially backing off each time until it gave up and fired up a new connection.

This could take up to 30 seconds or more to timeout the TCP retransmits and thus cause hang problems within the application whilst the retransmissions take place.

We used to fix this problem by either setting Windows to send a KeepAlive packet at an interval lower than the Firewall's idle timeout value, or adjust the firewall settings.

In my experience with this behaviour I believe it's likely to cause the following, but not limited to the following problems:

Disconnect pop ups in Outlook
Unexpected authentication prompts
Hangs within Outlook where we get a 'polo mint' especially when switching mailboxes/calendars.
Performance problems
Mail stuck in outbox for an extended period

• Ensure Outlook is bypassing the proxy server when connecting to Office 365.

Exceptions for Microsoft Office 365 URLs and applications from the authentication proxy:

Allow outbound connections to the following destination: *.microsoftonline.com
Allow outbound connections to the following destination: *.microsoftonline-p.com
Allow outbound connections to the following destination: *.sharepoint.com
Allow outbound connections to the following destination: *.outlook.com
Allow outbound connections to the following destination: *.lync.com
Allow outbound connections to the following destination: osub.microsoft.com

Ports 80/443
Protocols TCP and HTTPS
Rule must apply to all users.
HTTPS/SSL time-out set to 8 hours.

How to harden the TCP/IP stack against denial of service attacks in Windows Server 2003
How to troubleshoot non-browser apps that can’t sign in to Office 365, Azure, or Intune
Network Perimeters & TCP Idle session settings for Outlook on Office 365
Outlook may take two to three minutes to connect to an Office 365 mailbox
Outlook takes several minutes to connect to Exchange Online (O365)
Preventing proxy authentication from delaying your O365 connection
Things that you may want to know about TCP Keepalives
Troubleshooting long running MAPI connections to Exchange Server 2010 through Network Load Balancers

Office 365 filtering:

• Always use URLs as IPs may change without notice.

Office 365 URL based filtering is just better and easier to sustain


Office 365 and non-Office 365 datacentres, and Office 365 edge nodes (local cache):


More info:

Accelerating Office 365 with SteelHead SaaS
Fast Track Network Analysis (Asia Pacific)
Network peering
Network planning and performance tuning for Office 365
Office 365 Network Topology and Performance Planning
Office 365 Performance Management
Office 365 URLs and IP address ranges
Top 10 Tips for Optimising & Troubleshooting your Office 365 Network Connectivity

1 comment:

  1. Thank you very much for sharing such a useful article. Will definitely saved and revisit your site INTELIGENCIA ARTIFICIAL APLICADA A LA EMPRESA - SEPE 2020

    ReplyDelete