Microsoft Discloses Cause of Massive Outlook Outage: Flaw Found in July Update Code

Photo of author

By Global Team

Microsoft has disclosed the cause of last week’s widespread outage affecting Outlook and Microsoft 365 services around the world. A potentially faulty server code update deployed in July combined with increased network traffic and reduced server capacity, triggering a chain of failures.

Charles Lamanna, Microsoft’s corporate vice president for Copilot, Agents, and Platform, explained the cause of the outage on Nov. 12 and the recovery process in a post on X on Nov. 16, local time, and issued an apology. “We deeply apologize for the impact on all our customers and recognize the disruption to their work and operations,” he said.

The outage disrupted Outlook email and calendar features, along with other Microsoft 365 services. Communication disruptions were also reported at government agencies, courts, hospitals, schools, and businesses. Microsoft said it found no evidence of malicious activity in connection with the incident.

Source: Microsoft
Source: Microsoft

Charles Lamanna, Microsoft’s corporate vice president for Copilot, Agents, and Platform, apologized on X on Nov. 16, local time, for the widespread Microsoft 365 outage that occurred on Nov. 12 and explained its cause. Microsoft said a potentially faulty update deployed in July, combined with increased server load and reduced available capacity, led to a global service disruption. (Image: Screenshot from Charles Lamanna’s X account)

According to Microsoft, the outage began in network gateway servers that handle user requests at the front end of Microsoft 365 services. The gateways receive external connection requests and route them to individual services such as Outlook and Teams.

The problematic code was part of an update deployed specifically to gateway servers in July. Microsoft rolled it out gradually to servers around the world over several weeks, and detected no issues for more than three months.

The code contained a flaw that, under certain load conditions, caused internal server processes to wait while competing for the same resources—a condition known as lock contention. As contention intensified, requests were delayed and servers could stop responding.

Pre-deployment load testing did not identify the flaw. Microsoft said the server specifications and traffic patterns in the test environment differed from those in production. In particular, lock contention can worsen on newer servers with more processing cores, but the tests had focused on older equipment with relatively few cores.

Lamanna said the flaw and update affected only Microsoft 365 gateway servers and did not apply to general Windows operating system deployments.

Increased Teams traffic and reduced server capacity combined

The potential flaw turned into an actual outage when server load increased while available capacity was reduced.

Microsoft operates two separate network gateway systems for Microsoft 365. If one system has a problem, services are designed to switch to the other gateway to limit the impact.

About 15 hours before the incident, Teams traffic was shifted from the backup gateway to the primary gateway. The primary gateway also handled Exchange Online, which supports Outlook email. The additional Teams traffic increased server load and changed the pattern of incoming requests.

At the time, about 14% of gateway server capacity was offline because of a separate issue the previous day. Microsoft was restoring that capacity through routine maintenance and recovery procedures, and said the reduced capacity alone was not enough to affect services.

The combination of increased traffic and reduced server capacity triggered the flaw in the July update. User requests began to pile up, and servers stopped responding. The outage occurred during a period of peak service use in the United States.

Automatic traffic rerouting then contributed to the spread of the outage. When a server connection fails, Microsoft’s systems automatically redirect user requests to healthy servers. In this case, requests shifted away from failed servers overwhelmed the remaining servers, triggering the same flaw in succession.

Microsoft said the problem had spread across gateway servers within two hours of the initial failure. At the height of the outage, more than 80% of all gateway servers were unresponsive.

Isolation mechanisms meant to contain the outage also failed

Microsoft divides its network gateways into units with capacity for thousands of servers and manages them by region. The system is designed to keep a problem in one area from spreading to others.

In this incident, a separate issue prevented the isolation mechanisms designed to stop failures from spreading between areas from working as intended. The chain reaction that began on the servers crossed regional boundaries and spread across the global gateway infrastructure.

Recovery times varied by service. Teams, OneDrive, SharePoint, and Office apps recovered relatively quickly because they could switch from the affected gateways to other routes. Teams chat and meeting features worked normally during most of the outage.

Exchange Online could not switch to another route from the affected gateways, so its email disruption lasted longer.

Microsoft: Final report to be released this week

Problems continued during the recovery process. When engineers restarted servers, a surge of unprocessed user requests caused them to become unresponsive again.

Existing automatic recovery procedures were not designed for a situation in which such a large number of servers stopped responding at once. Because the faulty code had been deployed months earlier, it was also difficult to quickly reverse through standard change-rollback procedures or feature controls.

After identifying the cause, Microsoft developed an update to restore the code path to its previous, stable state. It then took servers out of service in batches, applied the fix, restarted them, and returned them to operation. Some recovery work involved more than half the server capacity in a single region.

Microsoft has posted details about the incident in the Microsoft 365 admin center and plans to release a final incident review report this week.